Handle image templates that don't use zero padding #705

dagewa · 2024-03-08T15:18:32Z

Following the discussion in #646, a template with a single # character is used to expand to all image files with non-zero-padded sequential numbers.

codecov · 2024-03-08T15:29:17Z

Codecov Report

Attention: Patch coverage is 86.11111% with 5 lines in your changes are missing coverage. Please review.

Project coverage is 42.26%. Comparing base (6818056) to head (fc5f8f4).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #705      +/-   ##
==========================================
+ Coverage   42.23%   42.26%   +0.02%     
==========================================
  Files         187      187              
  Lines       16752    16784      +32     
  Branches     3193     3201       +8     
==========================================
+ Hits         7075     7093      +18     
- Misses       9031     9045      +14     
  Partials      646      646

dagewa · 2024-03-16T17:11:44Z

Sorry, there's a bunch of extra commits in here. I think that's because I merged main from the wrong remote at some point 😞

dagewa · 2024-03-17T10:27:06Z

Improved specificity further by ensuring the new pattern is only matched when the image serial number does not start with zero. All tests are passing for me, but this code is knotty because not only are we using regexes, but were using reversed regexes 😱

…zero-padded incremental number.

for the case of non-zero padded indices that extend to multiple digits. Other use cases should still work the same way.

make a new template that includes the underscore character, as that appears to be part of the standard Rigaku CAP filename format.

The non-zero-padded template must not start with `0` in the image number, so add that to the template. Then there are 4 groups, so add a condition to handle this case separately.

…ero-padding

dagewa · 2024-03-22T14:11:25Z

Finally got the CI all happy. Locally, dxtbx and DIALS tests all pass, as do xia2 tests run with pytest --regression -n auto.

Has anybody got any comments on this PR?

graeme-winter

Looks sensible

I feel like we should have some test code where we pass a list of filenames and we get back a list which matches the behaviour, more as documentation for future developers than for actual testing.

I would ask that we comment in the change log how you would handle wanting to only open single digit files. I specifically wonder about wanting to dials.import template=foo_001#.cbf say which may not do what you expect (though, probably would)

src/dxtbx/imageset.py

newsfragments/705.feature

src/dxtbx/model/scan_helpers.py

src/dxtbx/sequence_filenames.py

dagewa · 2024-03-27T15:39:18Z

I would ask that we comment in the change log how you would handle wanting to only open single digit files. I specifically wonder about wanting to dials.import template=foo_001#.cbf say which may not do what you expect (though, probably would)

I believe this case does do the right thing:

$ dials.import template=$(dials.data get -q insulin)/insulin_1_01#.img
DIALS (2018) Acta Cryst. D74, 85-97. https://doi.org/10.1107/S2059798317017235
DIALS 3.dev.1103-gea4fc09a8
The following parameters have been modified:

input {
  template = "/home/fcx32934/sw/cctbx/build/dials_data/insulin/insulin_1_01#.img"
}

--------------------------------------------------------------------------------
  format: <class 'dxtbx.format.FormatSMVADSCSN.FormatSMVADSCSN'>
  template: /home/fcx32934/sw/cctbx/build/dials_data/insulin/insulin_1_01#.img:0:9
  num images: 10
  sequences:
    still:    0
    sweep:    1
  num stills: 0
--------------------------------------------------------------------------------
Writing experiments to imported.expt

The dials.show imported.expt output shows that

Scan:
    number of images:   10
    image range:   {0,9}

Checking the image viewer shows that the first image is indeed the expected image.

dagewa · 2024-03-27T16:05:56Z

Note to self: one thing to check is what happens with non-zero-padded series that do not start with a single digit. For example:

image_98.cbf
image_99.cbf
image_100.cbf

dagewa · 2024-03-27T16:46:14Z

Ok, seems to work fine. With files named insulin_1_{98..101}.img, implicitly:

$ dials.import insulin_1_*
DIALS (2018) Acta Cryst. D74, 85-97. https://doi.org/10.1107/S2059798317017235
DIALS 3.dev.1103-gea4fc09a8
The following parameters have been modified:

input {
  experiments = <image files>
}

--------------------------------------------------------------------------------
  format: <class 'dxtbx.format.FormatSMVADSCSN.FormatSMVADSCSN'>
  template: /tmp/insulin_1_##.img:98:101
  num images: 4
  sequences:
    still:    0
    sweep:    1
  num stills: 0
--------------------------------------------------------------------------------

and with an explicit template:

$ dials.import template=insulin_1_#.img
DIALS (2018) Acta Cryst. D74, 85-97. https://doi.org/10.1107/S2059798317017235
DIALS 3.dev.1103-gea4fc09a8
The following parameters have been modified:

input {
  template = "insulin_1_#.img"
}

--------------------------------------------------------------------------------
  format: <class 'dxtbx.format.FormatSMVADSCSN.FormatSMVADSCSN'>
  template: insulin_1_#.img:98:101
  num images: 4
  sequences:
    still:    0
    sweep:    1
  num stills: 0
--------------------------------------------------------------------------------

both produce a 4-image sweep.

…all the added code

Don't include the directory path in the regex, because a pattern starting with "C:\\Users" will result in: re.error: incomplete escape \U at position 2 Instead, only do the regex on the filename part.

dagewa · 2024-04-18T08:36:00Z

Added test was useful: unearthed an error on Windows, now fixed 👍 I believe this is ready to go in.

Update test command following cctbx/dxtbx#705

* Use a single `#` character in a filename template to represent a non-zero-padded incremental number.

dagewa linked an issue Mar 8, 2024 that may be closed by this pull request

Handling image templates that don't use zero-padding #646

Closed

dagewa requested review from graeme-winter and biochem-fan March 8, 2024 15:20

dagewa mentioned this pull request Mar 15, 2024

Handling image templates that don't use zero-padding #646

Closed

dagewa changed the title ~~Handle image templates that dont use zero padding~~ Handle image templates that don't use zero padding Mar 15, 2024

ndevenish force-pushed the 646-handling-image-templates-that-dont-use-zero-padding branch from e4026c5 to 45e88b2 Compare March 21, 2024 10:05

dagewa and others added 7 commits March 21, 2024 10:11

Use a single # character in a filename template to represent a non-…

9f21c85

…zero-padded incremental number.

News

c7425be

Rename newsfragments/xxx.feature to newsfragments/705.feature

0d73579

Extend the single-digit template reader added in 339dfd7 to work

e6a97d3

for the case of non-zero padded indices that extend to multiple digits. Other use cases should still work the same way.

Change tactic slightly: rather than extend the single digit template,

4f10bef

make a new template that includes the underscore character, as that appears to be part of the standard Rigaku CAP filename format.

Improve specificity further.

fbabb9f

The non-zero-padded template must not start with `0` in the image number, so add that to the template. Then there are 4 groups, so add a condition to handle this case separately.

Include in test_template_regex

7dee593

ndevenish force-pushed the 646-handling-image-templates-that-dont-use-zero-padding branch from 45e88b2 to 7dee593 Compare March 21, 2024 10:14

Merge branch 'main' into 646-handling-image-templates-that-dont-use-z…

f40a0ce

…ero-padding

graeme-winter approved these changes Mar 27, 2024

View reviewed changes

dagewa added 2 commits March 27, 2024 15:27

More descriptive function name

0edaf79

Make comment a comment, not a link

5fb4010

dagewa added 3 commits April 16, 2024 16:46

Add tests for locate_files_matching_template_string, which exercises …

6303168

…all the added code

More detailed newsfragment, as requested

10b7715

Avoid regex problem on Windows

fc5f8f4

Don't include the directory path in the regex, because a pattern starting with "C:\\Users" will result in: re.error: incomplete escape \U at position 2 Instead, only do the regex on the filename part.

dagewa mentioned this pull request Apr 17, 2024

image= mode for non-zero-padded image numbers xia2/xia2#788

Open

dagewa merged commit 24a50c7 into cctbx:main Apr 18, 2024
13 checks passed

jbeilstenedmands added a commit to xia2/xia2 that referenced this pull request Apr 19, 2024

Update test command following cctbx/dxtbx#705

598c02e

jbeilstenedmands mentioned this pull request Apr 19, 2024

Update xia2.ssx test following dxtbx template handling change xia2/xia2#790

Merged

jbeilstenedmands added a commit to xia2/xia2 that referenced this pull request Apr 19, 2024

Update xia2.ssx test following dxtbx template handling change (#790)

30f5da9

Update test command following cctbx/dxtbx#705

toastisme pushed a commit to toastisme/dxtbx that referenced this pull request Jul 18, 2024

Handle image templates that don't use zero padding (cctbx#705)

0eb9ff3

* Use a single `#` character in a filename template to represent a non-zero-padded incremental number.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle image templates that don't use zero padding #705

Handle image templates that don't use zero padding #705

dagewa commented Mar 8, 2024

codecov bot commented Mar 8, 2024 •

edited

Loading

dagewa commented Mar 16, 2024

dagewa commented Mar 17, 2024

dagewa commented Mar 22, 2024

graeme-winter left a comment

dagewa commented Mar 27, 2024

dagewa commented Mar 27, 2024

dagewa commented Mar 27, 2024

dagewa commented Apr 18, 2024

Handle image templates that don't use zero padding #705

Handle image templates that don't use zero padding #705

Conversation

dagewa commented Mar 8, 2024

codecov bot commented Mar 8, 2024 • edited Loading

Codecov Report

dagewa commented Mar 16, 2024

dagewa commented Mar 17, 2024

dagewa commented Mar 22, 2024

graeme-winter left a comment

Choose a reason for hiding this comment

dagewa commented Mar 27, 2024

dagewa commented Mar 27, 2024

dagewa commented Mar 27, 2024

dagewa commented Apr 18, 2024

codecov bot commented Mar 8, 2024 •

edited

Loading