Subarray trigger does not match telescopes with data in prod6 files

[maxnoe]

This breaks some assumptions in SimTelEventSource / DataWriter / TableLoader / HDF5EventSource and results in errors
since the length of the trigger table does not match the length of the dl1 / dl2 tables.

Investigating why this happens, I think this might be related to RANDOM_MONO feature of sim_telarray active in prod6

[maxnoe]

In the output file of sim_telarray, the event that causes this mismatch is "weird".

Here is the structure:

 MCEvent[2021](event_id=1548602)
 TelescopeData[1204](event_id=1548602)
     PhotoElectrons[1208](array_id=2, telescope_id=0)
     PhotoElectrons[1208](array_id=2, telescope_id=1)
     PhotoElectrons[1208](array_id=2, telescope_id=2)
     PhotoElectrons[1208](array_id=2, telescope_id=3)
     PhotoElectrons[1208](array_id=2, telescope_id=6)
     And 3 objects more of the same type
 MCPhotoelectronSum[2026](event_id=1548602)
 ArrayEvent[2010](event_id=1548602)
     TriggerInformation[2009](event_id=1548602)
     TelescopeEvent[2203](telescope_id=3)
         TelescopeEventHeader[2011](telescope_id=3)
         ADCSamples[2013](telescope_id=3)
         ImageParameters[2014](telescope_id=3)
         PixelList[2027](telescope_id=3, code=1, kind=selected)
     TelescopeEvent[2208](telescope_id=8)
         TelescopeEventHeader[2011](telescope_id=8)
         ADCSamples[2013](telescope_id=8)
         ImageParameters[2014](telescope_id=8)
         PixelList[2027](telescope_id=8, code=0, kind=triggered)
         PixelList[2027](telescope_id=8, code=1, kind=selected)
     TrackingPosition[2101](telescope_id=1)
     TrackingPosition[2102](telescope_id=2)
     TrackingPosition[2103](telescope_id=3)
     TrackingPosition[2104](telescope_id=4)
     TrackingPosition[2105](telescope_id=5)
     And 26 objects more of the same type
     StereoReconstruction[2015](result_bits=000000000101)

and the parsed trigger information:

In [5]: e["trigger_information"]
Out[5]: 
{'cpu_time': (1708711789, 292339000),
 'gps_time': (1708711789, 292339000),
 'trigger_pattern': 4,
 'data_pattern': 4,
 'n_triggered_telescopes': 1,
 'triggered_telescopes': array([3], dtype=int16),
 'trigger_times': array([10.260431], dtype=float32),
 'n_telescopes_with_data': 1,
 'telescopes_with_data': array([3], dtype=int16),
 'teltrg_type_mask': array([66], dtype=uint64),
 'teltrg_time_by_type': {},
 'plane_wavefront_compensation': {'az': 0.0,
  'alt': 1.2217304706573486,
  'speed_of_light': 29.972414016723633}}

So telescopes_with_data / n_telescopes_with_data doesn't match the reality, there are 2 telescope events (tel_ids 3 and 8):

Here are the two telescope event headers:

In [8]: e["telescope_events"][3]["header"]
Out[8]: 
{'loc_count': 443,
 'glob_count': 1548602,
 'cpu_time': (1708711789, 292339000),
 'gps_time': (0, 0),
 'trg_source': 1,
 'list_trgsect': array([332, 333, 337, 380, 384, 385]),
 'time_trgsect': array([41.259766, 41.503906, 41.503906, 41.748047, 41.503906, 41.748047],
       dtype=float32),
 'readout_time': 293.5289001464844,
 'relative_trigger_time': 9.033203125,
 'start_readout': 33,
 'telescope_id': 3}

In [9]: e["telescope_events"][8]["header"]
Out[9]: 
{'loc_count': 208,
 'glob_count': 1548602,
 'cpu_time': (1708711789, 292339000),
 'gps_time': (0, 0),
 'trg_source': 1,
 'list_trgsect': array([ 910,  911,  913,  920,  921,  922,  923, 1040, 1041]),
 'time_trgsect': array([56.25, 55.75, 56.5 , 56.  , 57.  , 55.5 , 55.75, 56.25, 56.75],
       dtype=float32),
 'readout_time': 168.71551513671875,
 'relative_trigger_time': 18.5,
 'start_readout': 37,
 'telescope_id': 8}

@kbernloehr @orelgueta Any idea what is going on here?

File is the same as in #2344: /vo.cta.in2p3.fr/MC/PROD6/LaPalma/gamma/sim_telarray/4749/Data/000xxx/gamma_20deg_0deg_run000065___cta-prod6-2156m-LaPalma-dark+magic.simtel.zst

[maxnoe]

This happens four times in the file, every time the stereo trigger information contains a single telescope, but then there are two telescope events:

2024-11-28 13:41:02,634 INFO [ctapipe.ctapipe-process] (tool.initialize): Loading config from '[]'
2024-11-28 13:41:02,635 INFO [ctapipe.ctapipe-process] (tool.initialize): ctapipe version 0.23.1.dev5+g05e5453d.d20241128
2024-11-28 13:41:02,636 INFO [ctapipe.ctapipe-process.SimTelEventSource] (eventsource.__init__): INPUT PATH = /home/maxnoe/Uni/CTA/data/grid-errors/gamma_20deg_0deg_run000065___cta-prod6-2156m-LaPalma-dark+magic.simtel.zst
2024-11-28 13:41:03,071 WARNING [ctapipe.ctapipe-process.DataWriter] (datawriter._setup_output_path): Overwriting /home/maxnoe/Uni/CTA/data/grid-errors/test.dl1.h5
2024-11-28 13:41:03,212 INFO [ctapipe.ctapipe-process] (process.start): applying calibration: True
2024-11-28 13:41:03,212 INFO [ctapipe.ctapipe-process] (process.start): (re)compute DL1: True
2024-11-28 13:41:03,212 INFO [ctapipe.ctapipe-process] (process.start): (re)compute DL2: False
2024-11-28 13:41:03,212 INFO [ctapipe.ctapipe-process] (process.start): compute muon parameters: False
2024-11-28 13:41:03,212 INFO [ctapipe.ctapipe-process] (subarray.info): Subarray : LaPalma-prod6
2024-11-28 13:41:03,212 INFO [ctapipe.ctapipe-process] (subarray.info): Num Tels : 32
2024-11-28 13:41:03,213 INFO [ctapipe.ctapipe-process] (subarray.info): Footprint: 0.65 km2
2024-11-28 13:41:03,213 INFO [ctapipe.ctapipe-process] (subarray.info): Height   : 2156.00 m
2024-11-28 13:41:03,214 INFO [ctapipe.ctapipe-process] (subarray.info): Lon/Lat  : -17.892030200000022 deg, 28.762166100000005 deg 
2024-11-28 13:41:03,214 INFO [ctapipe.ctapipe-process] (subarray.info): 
2024-11-28 13:41:03,235 INFO [ctapipe.ctapipe-process] (subarray.info):        Type       Count Tel IDs
2024-11-28 13:41:03,235 INFO [ctapipe.ctapipe-process] (subarray.info): ----------------- ----- -------
2024-11-28 13:41:03,235 INFO [ctapipe.ctapipe-process] (subarray.info):    LST_LST_LSTcam     1 1      
2024-11-28 13:41:03,235 INFO [ctapipe.ctapipe-process] (subarray.info): LST_MAGIC1_MAGIC1     1 31     
2024-11-28 13:41:03,235 INFO [ctapipe.ctapipe-process] (subarray.info): LST_MAGIC2_MAGIC2     1 32     
2024-11-28 13:41:03,235 INFO [ctapipe.ctapipe-process] (subarray.info): MST_MST_NectarCam    26 5-30   
2024-11-28 13:41:56,892 WARNING [ctapipe.ctapipe-process.SimTelEventSource] (simteleventsource._generate_events): Encountered telescope event not present in stereo trigger information, skipping. event_id = 1548602, tel_id = 8, tels_with_trigger: [3]
2024-11-28 13:42:00,431 INFO [ctapipe.ctapipe-process.SimTelEventSource] (simteleventsource._generate_events): telescope event event_id = 1664106, tel_id = 32 is missing true image
2024-11-28 13:42:21,941 WARNING [ctapipe.ctapipe-process.SimTelEventSource] (simteleventsource._generate_events): Encountered telescope event not present in stereo trigger information, skipping. event_id = 2247909, tel_id = 14, tels_with_trigger: [32]
2024-11-28 13:43:17,227 WARNING [ctapipe.ctapipe-process.SimTelEventSource] (simteleventsource._generate_events): Encountered telescope event not present in stereo trigger information, skipping. event_id = 3974908, tel_id = 5, tels_with_trigger: [2]
2024-11-28 13:43:44,533 WARNING [ctapipe.ctapipe-process.SimTelEventSource] (simteleventsource._generate_events): Encountered telescope event not present in stereo trigger information, skipping. event_id = 4839806, tel_id = 22, tels_with_trigger: [1]
2024-11-28 13:43:48,932 INFO [ctapipe.ctapipe-process] (tool.run): Finished: ctapipe-process
2024-11-28 13:43:48,934 INFO [ctapipe.ctapipe-process] (tool.write_provenance): Output: 
2024-11-28 13:43:48,936 INFO [ctapipe.ctapipe-process.DataWriter] (datawriter.finish): Finishing output

[maxnoe]

@kbernloehr Might this be the case where two telescopes trigger RANDOM_MONO in the same array event?