Flash data discussion

[zain-sohail]

#459 after discussion, I was looking at the raw data and in the electron channels, there are some pulseIds that shouldn't exist. The h5 file contains them so it's not an sed issue but should be figured out and address in the loading step.
This is from the tutorial dataset Gd_W110 filename FLASH1_USER3_stream_2_run44799_file1_20230323T005636.1.h5 filtered to see pulses above 1000

[rettigl]

These could maybe just be dark counts? How is the DLD triggered, how many microbunch triggers are being sent in-between train resets? You can filter them out during loading, but I suppose this is not necessary, it anyways should not be related to #459

[zain-sohail]

it anyways should not be related to #459

Yes, I didn't mean that they were related, just that the discussion made me look at the raw data in more detail.

These could maybe just be dark counts? How is the DLD triggered, how many microbunch triggers are being sent in-between train resets? You can filter them out during loading, but I suppose this is not necessary,

But yes that makes sense. I'd ping @kutnyakhov for this.

[zain-sohail]

I will move this to discussion.

Dima plans to do a run without source to double check if it's just dark counts.

Eitherways we can discuss later if these need to be filtered out.

[zain-sohail]

@kutnyakhov will check this

[zain-sohail]

Another issue I noticed. Currently we fill the train and pulse channels values between runs as well. However, this would lead to unintended data.
As unless the new run starts with the train data values, it would fill the previous run values which would be incorrect if time span between runs is long.
Should be checked.