Bug somewhere in SNR calculation

[wbenoit26]

Four of the injected events in the recent O4b run had SNRs between 1e10 and 1e18. Their parameters, below, all look normal. I'll try to recreate the bug using these values.

Our network put all of these events well into the background distribution, which makes sense, and these outliers don't affect the results at all.

a_1 [0.03214588 0.65104015 0.33481056 0.18206648]
a_2 [0.0354525  0.7128843  0.45888562 0.31506946]
dec [-0.47128954  0.3272408  -0.50630942 -1.12267578]
ifo_snrs [[2.89982710e+10 2.23091384e+10]
 [2.08057352e+11 5.22145997e+10]
 [7.02940520e+13 1.27724878e+14]
 [7.52688346e+17 1.04713996e+18]]
injection_time [1.41121931e+09 1.41214971e+09 1.41118244e+09 1.41219702e+09]
mass_1 [33.43138559 23.34850721  9.08087701 15.34435379]
mass_2 [28.52578271 13.59012262  8.36420156 15.14200361]
phase [6.13326988 6.23012628 4.88241533 5.77553712]
phi_12 [2.21459852 0.58514667 2.88959134 3.07214592]
phi_jl [0.87483594 4.03666391 2.06919499 3.10835041]
psi [2.02087484 0.76165215 1.25340103 0.96454069]
ra [3.61469318 2.82976046 4.14497266 4.5451069 ]
redshift [1.60790654 1.36705793 0.51883109 0.67943564]
shift [[0. 7.]
 [0. 6.]
 [0. 2.]
 [0. 2.]]
snr [3.65868483e+10 2.14509257e+11 1.45790584e+14 1.28958984e+18]
theta_jn [2.44985596 0.22330424 2.02124946 2.38520752]
tilt_1 [0.76561162 0.47777156 0.7685099  0.83465098]
tilt_2 [0.88908598 0.27688647 0.69188287 0.44033575]

[wbenoit26]

Actually, it looks like the waveforms themselves are way too large.

[wbenoit26]

It appears this is a known issue with IMRPhenomXPHM: https://git.ligo.org/reed.essick/rpo4-injection-triage/-/blob/main/rpo4a/README.md?ref_type=heads. Our rate of these instances is remarkably similar to what Reed reports for big_snr events, a few every 500,000.

It seems that this issue was avoided by turning off multi-banding: https://git.ligo.org/reed.essick/rpo4-injection-triage/-/blob/main/rpo4a/MULTIBANDING.md?ref_type=heads. We can't doing that without dramatically changing our waveform generation process, so I think a reasonable solution would be to simply have an upper SNR threshold of about 1000 during waveform generation.