PyCBC offline multi ifo coincident search workflow notes

Notes on executables for the multi-ifo search

See PyCBC-offline-multi-ifo-coincident-search for context & updated status of the pipeline.

How does the workflow creation script work

The "pycbc_create_offline_search_workflow" executable creates the workflow.

Lines 1 - 198 do the same thing as the old pycbc_make_coinc_search_workflow executable does (setting up the workflow, vetos and bank)

Lines 199 - 208 set up the inspiral jobs. The code works the same way as the old pycbc_make_coinc_search_workflow, but config file changes are required to see the 3rd detector. The config changes required can be found at https://github.com/gwastro/pycbc-config/tree/master/O3exp/pipeline

Lines 209 - 214 create the trigger fitting jobs, these were required for O2 statistic calculations. (This will proably change)

Lines 215 - 261: This part sets up the multi ifo coincidence search. Details:

• First the variables containing the output files and the dictionary that will contain the precedence list is created. (lines 219-223)
• Next we read in the precedence list from the config files in lines 226-228. For any given combination of ifos, time shifted coincidences are generated by shifting one ifo relative to all the rest The shifted ('pivot') ifo will be the one appearing first in this list
• Lines 232 - 233 set up all possible detector combinations. (viable for any number of detectors)
• Lines 236 - 238 get the index of each ifo from the precedence list
• Lines 241 - 244 get the trigger files for the specific detector combination only.
• Lines 247 - 250 set up the pivot and fixed ifos required by pycbc_multiifo_coinc_findtrigs based on the precedence indexes. The lowest index will be the pivot ifo, the second lowest will be the fixed ifo.
• Lines 253 - 260 set up the multiifo coinc jobs.

Lines 262 - 307 set up the psd and range calculation.

Line 308 - 349 set up injection jobs for all detectors, and merges the triggers. Coincidence calculation is not yet implemented for this.

Lines 350 - 401 finalize the workflow.

pycbc_multiifo_coinc_findtrigs

This executable is the modification of 'pycbc_coinc_findtrigs'. The differences are, that this file requires two new input: pivot and fixed ifos, and the option to set strict coincidence has been removed, the background is generated only from coincident times.

Everything is set up using dictionaries keyed by ifo. The number of trigger files determines the number of ifos.

The coinc.time_multi_coincidence determines the coincidence triggers. Full 2 detector coincidence is done between the pivot and fixed ifos. Additional detectors do not slide independently of the fixed one, each trigger in an additional detector must be coincident with an existing coincident one. All times are moved to 'fixed' relative time.

The rank_method.coinc_multiifo is the used ranking statistic, currently only newsnr is implemented. (new statistics will be required for O3)

Next steps

• The background from the 2 and 3 detector coincidence jobs need to be combined.
  • Proposal: get the triggers from all sources (4 in case of 3 detectors). The triggers that come from only 2 detector times are left alone, and all triggers that come from 4 files (3 detector times) get a trials factor (/4) applied (similarly to pycbc_combine_statmap). This would require a ranking statistic that takes into account which detectors are online at a given time. Also, clustering should be done after this step (need to remove it from pycbc_multiifo_coinc_findtrigs).
• Once we have a combined background, we need to get ifar values, that should be done similarly to pycbc_coinc_statmap, but it will need to be rewritten for the new background files and multiple detectors.
• Need to set up injection finding.