browser in state of paralyisis due to SKA-sized data set

[twillis449]

OK - so I have an SKA-sized data set 35 minutes long with 0.1 sec sampling and one channel, so about 19000 baselines with 21000 samples -> 400 million rows in the MS. I start up the meqbrowser and load turbo-sim.py. Despite not having a .tdl.conf file, turbo-sim seems to hunt for the first (and only) MS it can find, and then goes lot to lunch for the next 16 minutes while it seems to need to keep rereading and opening the MS with the standard sorts of comments

Using LSM module from Tigger (using svn version) at /usr/local/lib/python2.7/dist-packages/Tigger (in path)
Successful readonly open of default-locked table dummy.MS: 24 columns, 405426000 rows
blah blah ...

What's going on here? All these apparently repeat re-reads are taking forever ....!

[IanHeywood]

Hi Tony. Maybe look into Multi-MS. You can create virtual Measurement Sets on the disk that point to sub-regions of your master MS, and should be interpreted as if they were individual Measurement Sets. That way you don't have to worry about splitting and re-concatenation, your SKA-sized MS can remain intact and hopefully the chunks will be small enough for MeqTrees etc. to digest...

http://casa.nrao.edu/docs/cookbook/casa_cookbook011.html

[twillis449]

Thanks Ian - I'll try that - in any case I'm still having trouble with Calico-based selfcal and have been using the CASA gaincal and applycal tasks. Splitting an MS should certainly speed up those tasks.

[o-smirnov]

@twillis449 could you please post a copy of the console output. Is it just those two messages?

[twillis449]

Here's the entire output (you can generate the MS with the script make_ms_ska_batch.py in my test_fitting directory on jake). After generating the MS I start up the meqbrowser and load turbo-sim.py. The browser then takes 20 minutes !! before the selection GUI appears ..... I start up at 9:08 AM and GUI finally appears at 9:28 AM

[NRC-005592LX 9:08am] [iono_sims]> meqbrowser
Welcome to the MeqTrees Browser!
Please wait a second while the GUI starts up.
Binding to local socket =meqbrowser-1000:1
Binding to TCP port 5000, remote meqservers may connect with gwpeer=:5000
gui(meqserver_gui.py:589:_start_kernel): /home/twillis/trunk/Timba/install/current/bin/meqserver -mt 4
gui(meqserver_gui.py:600:_start_kernel): started meqserver process 4193
(Meow.MSUtils: found /usr/local/bin/lwimager, can use it for imaging.)
(Meow.MSUtils: found image viewer /home/twillis/trunk/tigger/Tigger/bin/tigger)
(Meow.MSUtils: found image viewer /usr/local/bin/kvis)
(Meow.MSUtils: found image viewer /usr/local/bin/ds9)
(Meow.MSUtils: found image viewer ~/Tigger/tigger)
Successful readonly open of autonoread-locked table dummy.MS: 24 columns, 405426000 rows
Successful readonly open of autonoread-locked table /media/disk/twillis1/iono_sims/dummy.MS/ANTENNA: 8 columns, 197 rows
Successful readonly open of default-locked table /media/disk/twillis1/iono_sims/dummy.MS/OBSERVATION: 9 columns, 7 rows
Successful readonly open of autonoread-locked table /media/disk/twillis1/iono_sims/dummy.MS/POLARIZATION: 4 columns, 1 rows
Successful readonly open of autonoread-locked table /media/disk/twillis1/iono_sims/dummy.MS/DATA_DESCRIPTION: 3 columns, 1 rows
Successful readonly open of autonoread-locked table /media/disk/twillis1/iono_sims/dummy.MS/SPECTRAL_WINDOW: 14 columns, 1 rows
Successful readonly open of autonoread-locked table /media/disk/twillis1/iono_sims/dummy.MS/FIELD: 9 columns, 1 rows
Using LSM module from Tigger (using svn version) at /usr/local/lib/python2.7/dist-packages/Tigger (in path)
Successful readonly open of default-locked table dummy.MS: 24 columns, 405426000 rows
Successful readonly open of default-locked table dummy.MS: 24 columns, 405426000 rows
Successful readonly open of default-locked table dummy.MS: 24 columns, 405426000 rows
Successful readonly open of default-locked table dummy.MS: 24 columns, 405426000 rows
(Meow.MSUtils: found /usr/local/bin/lwimager, can use it for imaging.)
(Meow.MSUtils: found image viewer /home/twillis/trunk/tigger/Tigger/bin/tigger)
(Meow.MSUtils: found image viewer /usr/local/bin/kvis)
(Meow.MSUtils: found image viewer /usr/local/bin/ds9)
(Meow.MSUtils: found image viewer ~/Tigger/tigger)
Successful readonly open of autonoread-locked table dummy.MS: 24 columns, 405426000 rows
Successful readonly open of autonoread-locked table /media/disk/twillis1/iono_sims/dummy.MS/ANTENNA: 8 columns, 197 rows
Successful readonly open of default-locked table /media/disk/twillis1/iono_sims/dummy.MS/OBSERVATION: 9 columns, 7 rows
Successful readonly open of autonoread-locked table /media/disk/twillis1/iono_sims/dummy.MS/POLARIZATION: 4 columns, 1 rows
Successful readonly open of autonoread-locked table /media/disk/twillis1/iono_sims/dummy.MS/DATA_DESCRIPTION: 3 columns, 1 rows
Successful readonly open of autonoread-locked table /media/disk/twillis1/iono_sims/dummy.MS/SPECTRAL_WINDOW: 14 columns, 1 rows
Successful readonly open of autonoread-locked table /media/disk/twillis1/iono_sims/dummy.MS/FIELD: 9 columns, 1 rows
Using LSM module from Tigger (using svn version) at /usr/local/lib/python2.7/dist-packages/Tigger (in path)
Successful readonly open of default-locked table dummy.MS: 24 columns, 405426000 rows
Successful readonly open of default-locked table dummy.MS: 24 columns, 405426000 rows
Successful readonly open of default-locked table dummy.MS: 24 columns, 405426000 rows
Successful readonly open of default-locked table dummy.MS: 24 columns, 405426000 rows
^z
[NRC-005592LX 9:28am][iono_sims]> bg
[1] meqbrowser &

[twillis449]

By the way - a simple python script to just open the measurement set and load the entire CORRECTED_DATA column into memory on my 16 GB memory laptop only takes 2 min 44 sec.