Hit indices

[dermen]

I originally opened dials#198 , but the base repository was wrong!

So, in summary, this PR is for reprocessing XTC streams when you know the hit indices ahead of time. It works as expected when the psana datasource only includes a single run, but when doing things in multi-run mode, e.g. ds = psana.Datasource("exp=cxily7829:run=21-26:idx"), the indexing of events seems to get distorted. There are no "exceptions / code failures" from what I've seen, its just that the retrieved hits no longer correspond to what they were in the single run case. I honestly suspect its an issue with PSANA API not behaving as intended, most likely the bit where we get the event from the index, but it would be nice to trouble shoot for more eyes.

Long term, this could be useful if some preprocessing software or hardware determines the hit indices as data are rolling in from the detector.

[dermen]

Im open to meetings if folks are interested

[codecov]

Codecov Report

Attention: Patch coverage is 10.00000% with 18 lines in your changes are missing coverage. Please review.

Project coverage is 42.32%. Comparing base (78147cc) to head (e93cf4e).
Report is 6 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #709      +/-   ##
==========================================
+ Coverage   42.15%   42.32%   +0.16%     
==========================================
  Files         188      188              
  Lines       16764    16892     +128     
  Branches     3196     3245      +49     
==========================================
+ Hits         7067     7149      +82     
- Misses       9052     9093      +41     
- Partials      645      650       +5     

[phyy-nx]

Seems like a useful feature, thanks!

[dermen]

@phyy-nx , looking for a way to debug/ reproduce the multi-run failure . There are XTC streams in xfel regression, right ?

[phyy-nx]

There are (for both Rayonix and CSPAD), but only single runs. If you want multiple runs you gotta go to a deposition.

Mmmm, actually you might be able to cheat by copying the xfel_regression runs and renaming the files. For example the Rayonix run is run 20 of experiment mfxo1916. If you copy the xtc streams to a fake run 21 it might be fine.

For reference, this script will put the right folders in your path for psana to use the xfel_regression data.

[jbeilstenedmands]

Is there a way we can make this more of a general feature in dxtbx/dials.import? I am also interested in this, and I don't think it should be limited to xtc stream data?

[jbeilstenedmands]

We discussed this a bit in today's dials meeting - I just wanted to clarify that my comment was just intended to generate discussion and show my interest in these kind of ideas. I do not wish to hold up this PR or ask you to make a general solution, that would be a lot of work! I will make a separate issue to discuss this more generally idea as I think it is really useful. Thanks!

[dermen]

@jbeilstenedmands , totally agree!