Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Averaging issue in misaligned coarse channels #162

Open
d3v-null opened this issue Oct 11, 2024 · 2 comments
Open

Averaging issue in misaligned coarse channels #162

d3v-null opened this issue Oct 11, 2024 · 2 comments

Comments

@d3v-null
Copy link
Collaborator

When preprocessing an observation where a coarse channel ends early, birli is not correctly averaging the final timestamp on that coarse channel.

we'll use mwa-demo to test this

git clone https://github.com/mwaTelescope/mwa-demo
cd mwa-demo
mkdir -p demo/data/1128352792/raw
curl -L -o demo/data/1128352792/raw/1128352792_20151008151937_gpubox07_00.fits 'https://projects.pawsey.org.au/birli-test/1128352792_20151008151937_gpubox07_00.fits'
curl -L -o demo/data/1128352792/raw/1128352792_20151008151937_gpubox08_00.fits 'https://projects.pawsey.org.au/birli-test/1128352792_20151008151937_gpubox08_00.fits'

birli preprocess only first antenna and last few timesteps, no averaging (2s)

mkdir -p demo/data//1128352792/prep/
birli \
  -m demo/data/1128352792/raw/1128352792.metafits \
  --sel-time 50 53 --sel-ants 0 --flag-edge-width 80 \
  -u demo/data//1128352792/prep/birli_2s_1128352792.uvfits \
  demo/data//1128352792/raw/1128352792_2*.fits
    Scheduled start:      2015-10-08 15:19:35.000 UTC, unix=1444317575.000, gps=1128352792.000, mjd=4951034375.000, lmst= 3.5080°, lmst2k= 3.3100°, lat2k=-26.7907°                                                                                                                             
    Scheduled end:        2015-10-08 15:21:27.000 UTC, unix=1444317687.000, gps=1128352904.000, mjd=4951034487.000, lmst= 3.9759°, lmst2k= 3.7783°, lat2k=-26.7907°                                                                                                                             
    Scheduled duration:   112.000s =  56 * 2.000s                                                                                                                                                                                                                                               
    Quack duration:       2.000s =   1 * 2.000s                                                                                                                                                                                                                                                 
    Output duration:      8.000s =   4 * 2.000s                                                                                                                                                                                                                                                 
    Scheduled Bandwidth:  30.720MHz =  24 *  32 * 40.000kHz                                                                                                                                                                                                                                     
    Output Bandwidth:     2.560MHz =        64 * 40.000kHz                                                                                                                                                                                                                                      
    Timestep details (all=56, provided=54, common=53, good=52, select=4, flag=4):                                                                                                                                                                                                               
            2015-10-08 UTC +  unix [s]        gps [s]         p  c  g  s  f                                                                                                                                                                                                                     
      ts0:      15:19:35.000  1444317575.000  1128352792.000  p  c        f                                                                                                                                                                                                                     
      ts1:      15:19:37.000  1444317577.000  1128352794.000  p  c  g                                                                                                                                                                                                                           
...
     ts50:      15:21:15.000  1444317675.000  1128352892.000  p  c  g  s                                                                                                                                                                                                                        
     ts51:      15:21:17.000  1444317677.000  1128352894.000  p  c  g  s                                                                                                                                                                                                                        
     ts52:      15:21:19.000  1444317679.000  1128352896.000  p  c  g  s                                                                                                                                                                                                                        
     ts53:      15:21:21.000  1444317681.000  1128352898.000  p        s  f                  

    Coarse channel details (metafits=24, provided=2, common=2, good=2, select=2, flag=22, ranges=1):
            gpu  corr  rec  cen [MHz]  p  c  g  s  f  range 
...
     cc16:    8     7  147   188.1600  p  c  g  s     0 (0-23) 
     cc17:    7     6  148   189.4400  p  c  g  s     0 (0-23) 
...

plot and export auto magnitudes to tsv

demo/04_ssins.py demo/data/1128352792/prep/birli_2s_1128352792.uvfits --no-diff --threshold 9999 --export-tsv

birli_2s_1128352792 auto spectrum

gps_time	 187.5350	 187.5750	 187.6150	 187.6550	 187.6950	 187.7350	 187.7750	 187.8150	 187.8550	 187.8950	 187.9350	 187.9750	 188.0150	 188.0550	 188.0950	 188.1350	 188.1750	 188.2150	 188.2550	 188.2950	 188.3350	 188.3750	 188.4150	 188.4550	 188.4950	 188.5350	 188.5750	 188.6150	 188.6550	 188.6950	 188.7350	 188.7750	 188.8150	 188.8550	 188.8950	 188.9350	 188.9750	 189.0150	 189.0550	 189.0950	 189.1350	 189.1750	 189.2150	 189.2550	 189.2950	 189.3350	 189.3750	 189.4150	 189.4550	 189.4950	 189.5350	 189.5750	 189.6150	 189.6550	 189.6950	 189.7350	 189.7750	 189.8150	 189.8550	 189.8950	 189.9350	 189.9750	 190.0150	 190.0550
1128352893.000	-0.000	-0.001	-0.002	-0.005	0.004	-0.003	0.002	-0.005	0.005	0.007	-0.003	-0.000	-0.003	-0.003	0.008	0.003	-0.001	-0.001	-0.007	-0.003	0.004	-0.008	0.003	-0.002	-0.001	-0.007	-0.004	-0.001	-0.002	0.002	-0.005	0.002	0.444	0.445	0.442	0.437	0.444	0.447	0.447	0.440	0.448	0.445	0.436	0.436	0.441	0.450	0.433	0.437	0.435	0.433	0.446	0.433	0.443	0.442	0.437	0.443	0.440	0.445	0.444	0.439	0.436	0.449	0.430	0.428
1128352895.000	-0.004	-0.001	0.002	0.008	-0.003	-0.006	-0.002	-0.000	0.002	0.002	-0.000	0.000	-0.006	0.007	-0.002	0.001	0.002	-0.007	0.003	0.006	-0.003	0.000	-0.000	-0.002	0.002	-0.000	-0.002	-0.005	0.006	0.004	0.008	0.001	0.435	0.434	0.445	0.442	0.433	0.437	0.442	0.442	0.440	0.451	0.443	0.438	0.438	0.428	0.445	0.438	0.433	0.436	0.442	0.447	0.432	0.445	0.443	0.438	0.444	0.444	0.442	0.441	0.444	0.435	0.447	0.453
1128352897.000	0.000	0.006	-0.003	-0.004	-0.002	0.002	0.000	0.004	-0.004	-0.006	0.005	0.000	0.005	-0.005	-0.008	-0.003	-0.001	-0.001	-0.003	0.002	-0.000	-0.000	-0.001	-0.000	0.005	-0.001	0.012	0.007	0.001	-0.005	0.004	-0.001	0.444	0.444	0.437	0.445	0.447	0.440	0.434	0.442	0.435	0.427	0.444	0.449	0.445	0.446	0.445	0.448	0.456	0.455	0.435	0.444	0.449	0.437	0.443	0.443	0.440	0.434	0.438	0.444	0.444	0.439	0.446	0.443
1128352899.000	0.004	-0.004	0.003	0.001	0.001	0.007	-0.000	0.001	-0.003	-0.003	-0.002	-0.000	0.005	0.001	0.002	-0.001	0.000	0.010	0.007	-0.005	-0.001	0.008	-0.002	0.004	-0.006	0.008	-0.005	-0.001	-0.005	-0.000	-0.007	-0.001	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324	-1.324

the last timestep on channel 148 is all negative, but that's fine because it's all flagged.

birli preprocess only first antenna and last few timesteps, averaging to 4s

birli \
  -m demo/data/1128352792/raw/1128352792.metafits \
  --sel-time 50 53 --sel-ants 0 --flag-edge-width 80 \
  --avg-time-res 4 \
  -u demo/data//1128352792/prep/birli_4s_1128352792.uvfits \
  demo/data//1128352792/raw/1128352792_2*.fits
demo/04_ssins.py demo/data/1128352792/prep/birli_4s_1128352792.uvfits --no-diff --threshold 9999 --export-tsv

birli_4s_1128352792 auto spectrum

gps_time	 187.5350	 187.5750	 187.6150	 187.6550	 187.6950	 187.7350	 187.7750	 187.8150	 187.8550	 187.8950	 187.9350	 187.9750	 188.0150	 188.0550	 188.0950	 188.1350	 188.1750	 188.2150	 188.2550	 188.2950	 188.3350	 188.3750	 188.4150	 188.4550	 188.4950	 188.5350	 188.5750	 188.6150	 188.6550	 188.6950	 188.7350	 188.7750	 188.8150	 188.8550	 188.8950	 188.9350	 188.9750	 189.0150	 189.0550	 189.0950	 189.1350	 189.1750	 189.2150	 189.2550	 189.2950	 189.3350	 189.3750	 189.4150	 189.4550	 189.4950	 189.5350	 189.5750	 189.6150	 189.6550	 189.6950	 189.7350	 189.7750	 189.8150	 189.8550	 189.8950	 189.9350	 189.9750	 190.0150	 190.0550
1128352894.000	-0.002	-0.001	0.002	0.002	0.001	-0.003	-0.000	-0.003	0.004	0.005	-0.003	-0.000	-0.005	0.004	0.006	0.002	0.000	-0.001	0.000	-0.001	0.000	-0.004	0.001	-0.001	-0.002	-0.002	-0.007	-0.005	0.000	0.004	0.002	0.001	0.440	0.440	0.003	-0.002	-0.003	0.001	0.004	-0.000	0.444	0.008	-0.002	-0.004	0.439	0.439	-0.002	-0.004	0.434	-0.008	0.003	0.440	0.437	0.443	0.440	0.441	0.442	0.445	0.002	0.440	0.440	0.001	0.439	0.441
1128352898.000	0.002	0.001	-0.002	-0.002	-0.001	0.003	0.000	0.003	-0.004	-0.005	0.003	0.000	0.005	-0.004	-0.006	-0.002	-0.000	0.001	-0.000	0.001	-0.000	0.004	-0.001	0.001	0.002	0.002	0.007	0.005	-0.000	-0.004	-0.002	-0.001	-0.440	-0.440	-0.003	0.002	0.003	-0.001	-0.004	0.000	-0.444	-0.008	0.002	0.004	-0.439	-0.439	0.002	0.004	-0.434	0.008	-0.003	-0.440	-0.437	-0.443	-0.440	-0.441	-0.442	-0.445	-0.002	-0.440	-0.440	-0.001	-0.439	-0.441

Now some negative values (not all) have made it into the last timestamp, and because not all of the visibilities that went into these were all flagged, these values are not flagged, they're just down-weighted.

Average should not be including these flagged values.
@d3v-null
Copy link
Collaborator Author

was not able to reproduce this issue in Marlu. MWATelescope/Marlu@b74adde#diff-c8aa32c8b3f0bfb2afad951e7427d2fa0e03a5218025e6afadaee1c07c9de949R434

@d3v-null
Copy link
Collaborator Author

might be an aoflagger thing, it only shows up with rfi enabled.

export time_res_s=4;
birli \
  --sel-time 50 53 \
  --sel-ants 0 \
  --avg-time-res $time_res_s \
  --avg-freq-res 80 \
  --flag-edge-width 0 \
  -m demo/data/1128352792/raw/1128352792.metafits \
  demo/data//1128352792/raw/1128352792_2*.fits \
  -u demo/data//1128352792/prep/birli_${time_res_s}s_80kHz_1128352792.uvfits \
&& demo/04_ssins.py --no-diff --threshold 9999 --export-tsv $_

birli_4s_80kHz_1128352792 auto spectrum

gps_time	 187.5550	 187.6350	 187.7150	 187.7950	 187.8750	 187.9550	 188.0350	 188.1150	 188.1950	 188.2750	 188.3550	 188.4350	 188.5150	 188.5950	 188.6750	 188.7550	 188.8350	 188.9150	 188.9950	 189.0750	 189.1550	 189.2350	 189.3150	 189.3950	 189.4750	 189.5550	 189.6350	 189.7150	 189.7950	 189.8750	 189.9550	 190.0350
1128352894.000	47849.730	47790.383	47580.281	47675.242	47718.297	47445.801	47126.418	46590.168	45567.430	45270.258	45027.156	44471.488	44114.973	44169.926	44952.309	46509.164	45658.797	45439.703	45166.535	45489.891	45577.422	45322.711	44957.766	44526.797	43751.695	43736.848	42851.535	42445.188	42365.641	42207.148	42911.750	44171.449
1128352898.000	48022.730	47645.055	47647.215	47793.059	47387.375	47573.438	46867.598	46307.125	45664.832	45274.172	44997.586	44495.887	44237.852	44579.625	44821.891	46410.742	45773.086	45434.098	45239.570	45228.828	45035.562	45534.949	22562.662	44734.473	44251.035	43521.223	21458.520	21254.477	21105.754	21093.191	42846.855	44288.273
birli --no-rfi \
  --sel-time 50 53 \
  --sel-ants 0 \
  --avg-time-res $time_res_s \
  --avg-freq-res 80 \
  --flag-edge-width 0 \
  -m demo/data/1128352792/raw/1128352792.metafits \
  demo/data//1128352792/raw/1128352792_2*.fits \
  -u demo/data//1128352792/prep/birli_${time_res_s}s_80kHz_1128352792.uvfits \
&& demo/04_ssins.py --no-diff --threshold 9999 --export-tsv $_

birli_4s_80kHz_1128352792 auto spectrum

gps_time	 187.5550	 187.6350	 187.7150	 187.7950	 187.8750	 187.9550	 188.0350	 188.1150	 188.1950	 188.2750	 188.3550	 188.4350	 188.5150	 188.5950	 188.6750	 188.7550	 188.8350	 188.9150	 188.9950	 189.0750	 189.1550	 189.2350	 189.3150	 189.3950	 189.4750	 189.5550	 189.6350	 189.7150	 189.7950	 189.8750	 189.9550	 190.0350
1128352894.000	47849.730	47790.383	47580.281	47675.242	47718.297	47445.801	47087.434	46590.168	45567.430	45270.258	44793.953	44471.445	44114.973	44169.926	44952.309	46469.344	45544.012	45439.703	45166.535	45491.965	45470.359	45322.711	44957.770	44526.797	43751.695	43538.270	42851.605	42445.207	42365.422	42205.293	42708.570	44171.449
1128352898.000	48022.730	47645.055	47647.215	47793.059	47387.375	47573.438	47126.207	46307.125	45664.832	45274.172	44831.289	44465.848	44237.852	44579.625	44821.891	46476.375	45650.793	45434.098	45239.570	45372.379	45081.391	45534.949	45125.328	44734.473	44251.035	43490.277	42917.289	42508.957	42211.570	42181.242	42733.941	44288.273

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@d3v-null