FormatROD #645

dagewa · 2023-07-20T08:55:42Z

This is for #11.

FormatROD.py consists of @biochem-fan's gist, edited to pass precommit checks and to change to a faster understand method.

This has been tested to work with a couple of images:

The conversation at #11 and FIXMEs in the code suggest that support remains partial, but it is a good start and I think worth adding to dxtbx at this point.

https://gist.githubusercontent.com/biochem-fan/5f76423120e92828f03a4763a622a1e9/raw/a16a5c223a2bb7ed4764dd720d26c6f55aef5b91/FormatROD.py

biochem-fan · 2023-07-20T09:01:00Z

I don't have time to study your changes at the moment.

I recommend that you test on https://xrda.pdbj.org/entry/93 as well, which has a more complicated geometry. At the moment this class does not read axes, so you cannot index all sweeps in one go but each sweep should be indexable individually.

dagewa · 2023-07-20T09:03:17Z

Thanks, I'll take a look 👍

codecov · 2023-07-20T09:41:56Z

Codecov Report

Merging #645 (f93f74c) into main (1625cfa) will decrease coverage by 0.29%.
Report is 2 commits behind head on main.
The diff coverage is 21.25%.

❗ Current head f93f74c differs from pull request most recent head 3ed0b46. Consider uploading reports for the commit 3ed0b46 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #645      +/-   ##
==========================================
- Coverage   39.39%   39.10%   -0.29%     
==========================================
  Files         178      179       +1     
  Lines       15519    15757     +238     
  Branches     2996     3040      +44     
==========================================
+ Hits         6113     6162      +49     
- Misses       8827     9012     +185     
- Partials      579      583       +4

dagewa · 2023-07-20T10:15:31Z

I confirm that after applying the file renaming workaround (#11 (comment)), all 14 datasets in XRD-93.zip can be indexed with the expected cell.

dagewa · 2023-07-20T10:21:17Z

I note the comment about porting decode_TY6_oneline into C++. I might look at that, but I don't think it is a blocker for this PR. Performance on those 14 datasets (which use the TY6 compression) seemed acceptable

biochem-fan

Now that you tested the code on several datasets, I think this is ready to be merged.
Thanks!

Non-essential suggestion:
How about mentioning the file name issue as a comment, or submiting an issue to the DIALS repository, so that someone who processes ROD datasets can find the workaround.

problems with dials.import.

dagewa · 2023-07-20T11:51:06Z

Thanks! I put the issue in dxtbx, because that's where the template handling is found. But I added a note to the module docstring too.

I'll wait to see if @graeme-winter has any other files to test before merging.

graeme-winter · 2023-07-26T10:25:22Z

Thanks! I put the issue in dxtbx, because that's where the template handling is found. But I added a note to the module docstring too.

I'll wait to see if @graeme-winter has any other files to test before merging.

Should Hypix 6000 data work?

Grey-Area work :) $ dials.import ../LY*
DIALS (2018) Acta Cryst. D74, 85-97. https://doi.org/10.1107/S2059798317017235
DIALS 3.dev.1007-g94744f537
/Users/graeme/git/dials/modules/dxtbx/src/dxtbx/format/FormatROD.py:232: RuntimeWarning: invalid value encountered in multiply
  gonio_angles = np.array(self._bin_header["start_angles_steps"]) * np.array(

The following parameters have been modified:

input {
  experiments = <image files>
}

--------------------------------------------------------------------------------
  format: <class 'dxtbx.format.FormatROD.FormatROD'>
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_#.rodhypix:1:1
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_##.rodhypix:10:10
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_###.rodhypix:100:109
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_##.rodhypix:11:11
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_###.rodhypix:110:119
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_##.rodhypix:12:12
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_###.rodhypix:120:129
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_##.rodhypix:13:13
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_###.rodhypix:130:139
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_##.rodhypix:14:14
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_###.rodhypix:140:149
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_##.rodhypix:15:15
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_###.rodhypix:150:159
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_##.rodhypix:16:16
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_###.rodhypix:160:169
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_##.rodhypix:17:17
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_###.rodhypix:170:179
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_##.rodhypix:18:18
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_###.rodhypix:180:189
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_##.rodhypix:19:19
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_###.rodhypix:190:199
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_#.rodhypix:2:2
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_##.rodhypix:20:20
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_###.rodhypix:200:200
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_##.rodhypix:21:29
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_#.rodhypix:3:3
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_##.rodhypix:30:39
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_#.rodhypix:4:4
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_##.rodhypix:40:49
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_#.rodhypix:5:5
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_##.rodhypix:50:59
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_#.rodhypix:6:6
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_##.rodhypix:60:69
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_#.rodhypix:7:7
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_##.rodhypix:70:79
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_#.rodhypix:8:8
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_##.rodhypix:80:89
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_#.rodhypix:9:9
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_1_##.rodhypix:90:99
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_#.rodhypix:1:1
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_##.rodhypix:10:10
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_###.rodhypix:100:109
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_##.rodhypix:11:11
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_###.rodhypix:110:119
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_##.rodhypix:12:12
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_###.rodhypix:120:129
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_##.rodhypix:13:13
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_###.rodhypix:130:139
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_##.rodhypix:14:14
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_###.rodhypix:140:149
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_##.rodhypix:15:15
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_###.rodhypix:150:159
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_##.rodhypix:16:16
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_###.rodhypix:160:169
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_##.rodhypix:17:17
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_###.rodhypix:170:179
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_##.rodhypix:18:18
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_###.rodhypix:180:189
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_##.rodhypix:19:19
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_###.rodhypix:190:199
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_#.rodhypix:2:2
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_##.rodhypix:20:20
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_###.rodhypix:200:200
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_##.rodhypix:21:29
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_#.rodhypix:3:3
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_##.rodhypix:30:39
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_#.rodhypix:4:4
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_##.rodhypix:40:49
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_#.rodhypix:5:5
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_##.rodhypix:50:59
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_#.rodhypix:6:6
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_##.rodhypix:60:69
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_#.rodhypix:7:7
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_##.rodhypix:70:79
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_#.rodhypix:8:8
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_##.rodhypix:80:89
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_#.rodhypix:9:9
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_2_##.rodhypix:90:99
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_#.rodhypix:1:1
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_##.rodhypix:10:10
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_###.rodhypix:100:109
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_##.rodhypix:11:11
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_###.rodhypix:110:119
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_##.rodhypix:12:12
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_###.rodhypix:120:129
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_##.rodhypix:13:13
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_###.rodhypix:130:139
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_##.rodhypix:14:14
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_###.rodhypix:140:149
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_##.rodhypix:15:15
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_###.rodhypix:150:159
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_##.rodhypix:16:16
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_###.rodhypix:160:169
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_##.rodhypix:17:17
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_###.rodhypix:170:179
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_##.rodhypix:18:18
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_###.rodhypix:180:189
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_##.rodhypix:19:19
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_###.rodhypix:190:199
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_#.rodhypix:2:2
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_##.rodhypix:20:20
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_###.rodhypix:200:200
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_##.rodhypix:21:29
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_#.rodhypix:3:3
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_##.rodhypix:30:39
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_#.rodhypix:4:4
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_##.rodhypix:40:49
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_#.rodhypix:5:5
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_##.rodhypix:50:59
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_#.rodhypix:6:6
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_##.rodhypix:60:69
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_#.rodhypix:7:7
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_##.rodhypix:70:79
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_#.rodhypix:8:8
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_##.rodhypix:80:89
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_#.rodhypix:9:9
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_3_##.rodhypix:90:99
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_#.rodhypix:1:1
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_##.rodhypix:10:10
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_###.rodhypix:100:109
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_##.rodhypix:11:11
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_###.rodhypix:110:119
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_##.rodhypix:12:12
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_###.rodhypix:120:129
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_##.rodhypix:13:13
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_###.rodhypix:130:139
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_##.rodhypix:14:14
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_###.rodhypix:140:149
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_##.rodhypix:15:15
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_###.rodhypix:150:159
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_##.rodhypix:16:16
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_###.rodhypix:160:169
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_##.rodhypix:17:17
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_###.rodhypix:170:179
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_##.rodhypix:18:18
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_###.rodhypix:180:189
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_##.rodhypix:19:19
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_###.rodhypix:190:199
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_#.rodhypix:2:2
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_##.rodhypix:20:20
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_###.rodhypix:200:200
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_##.rodhypix:21:29
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_#.rodhypix:3:3
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_##.rodhypix:30:39
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_#.rodhypix:4:4
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_##.rodhypix:40:49
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_#.rodhypix:5:5
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_##.rodhypix:50:59
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_#.rodhypix:6:6
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_##.rodhypix:60:69
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_#.rodhypix:7:7
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_##.rodhypix:70:79
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_#.rodhypix:8:8
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_##.rodhypix:80:89
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_#.rodhypix:9:9
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_4_##.rodhypix:90:99
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_#.rodhypix:1:1
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_##.rodhypix:10:10
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_###.rodhypix:100:109
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_##.rodhypix:11:11
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_###.rodhypix:110:119
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_##.rodhypix:12:12
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_###.rodhypix:120:129
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_##.rodhypix:13:13
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_###.rodhypix:130:139
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_##.rodhypix:14:14
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_###.rodhypix:140:149
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_##.rodhypix:15:15
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_###.rodhypix:150:159
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_##.rodhypix:16:16
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_###.rodhypix:160:169
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_##.rodhypix:17:17
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_###.rodhypix:170:179
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_##.rodhypix:18:18
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_###.rodhypix:180:189
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_##.rodhypix:19:19
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_###.rodhypix:190:199
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_#.rodhypix:2:2
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_##.rodhypix:20:20
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_###.rodhypix:200:200
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_##.rodhypix:21:29
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_#.rodhypix:3:3
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_##.rodhypix:30:39
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_#.rodhypix:4:4
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_##.rodhypix:40:49
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_#.rodhypix:5:5
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_##.rodhypix:50:59
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_#.rodhypix:6:6
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_##.rodhypix:60:69
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_#.rodhypix:7:7
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_##.rodhypix:70:79
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_#.rodhypix:8:8
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_##.rodhypix:80:89
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_#.rodhypix:9:9
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_5_##.rodhypix:90:99
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_#.rodhypix:1:1
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_##.rodhypix:10:10
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_###.rodhypix:100:109
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_##.rodhypix:11:11
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_###.rodhypix:110:119
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_##.rodhypix:12:12
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_###.rodhypix:120:129
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_##.rodhypix:13:13
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_###.rodhypix:130:139
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_##.rodhypix:14:14
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_###.rodhypix:140:149
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_##.rodhypix:15:15
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_###.rodhypix:150:159
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_##.rodhypix:16:16
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_###.rodhypix:160:169
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_##.rodhypix:17:17
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_###.rodhypix:170:179
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_##.rodhypix:18:18
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_###.rodhypix:180:189
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_##.rodhypix:19:19
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_###.rodhypix:190:199
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_#.rodhypix:2:2
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_##.rodhypix:20:20
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_###.rodhypix:200:200
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_##.rodhypix:21:29
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_#.rodhypix:3:3
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_##.rodhypix:30:39
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_#.rodhypix:4:4
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_##.rodhypix:40:49
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_#.rodhypix:5:5
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_##.rodhypix:50:59
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_#.rodhypix:6:6
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_##.rodhypix:60:69
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_#.rodhypix:7:7
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_##.rodhypix:70:79
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_#.rodhypix:8:8
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_##.rodhypix:80:89
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_#.rodhypix:9:9
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_6_##.rodhypix:90:99
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_#.rodhypix:1:1
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_##.rodhypix:10:10
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_###.rodhypix:100:109
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_##.rodhypix:11:11
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_###.rodhypix:110:119
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_##.rodhypix:12:12
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_###.rodhypix:120:129
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_##.rodhypix:13:13
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_###.rodhypix:130:139
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_##.rodhypix:14:14
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_###.rodhypix:140:149
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_##.rodhypix:15:15
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_###.rodhypix:150:159
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_##.rodhypix:16:16
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_###.rodhypix:160:169
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_##.rodhypix:17:17
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_###.rodhypix:170:179
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_##.rodhypix:18:18
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_###.rodhypix:180:189
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_##.rodhypix:19:19
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_###.rodhypix:190:199
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_#.rodhypix:2:2
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_##.rodhypix:20:20
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_###.rodhypix:200:200
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_##.rodhypix:21:29
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_#.rodhypix:3:3
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_##.rodhypix:30:39
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_#.rodhypix:4:4
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_##.rodhypix:40:49
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_#.rodhypix:5:5
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_##.rodhypix:50:59
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_#.rodhypix:6:6
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_##.rodhypix:60:69
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_#.rodhypix:7:7
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_##.rodhypix:70:79
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_#.rodhypix:8:8
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_##.rodhypix:80:89
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_#.rodhypix:9:9
  template: /Users/graeme/data/rigaku-hypix-6000/native/LY_HP6000_7_##.rodhypix:90:99
  num images: 1400
  sequences:
    still:    0
    sweep:    273
  num stills: 0
--------------------------------------------------------------------------------
Writing experiments to imported.expt

It does not appear to have recognised that _9.thing goes with _10.thing etc. - I thought that was in this PR? I saw commits along these lines

graeme-winter · 2023-07-26T10:28:04Z

OK, working around that; please hold the line

graeme-winter · 2023-07-26T10:59:22Z

OK, I have a 7-scan run, which only partly indexes:

+------------+-------------+---------------+-------------+
|   Imageset |   # indexed |   # unindexed | % indexed   |
|------------+-------------+---------------+-------------|
|          0 |       19254 |           199 | 99.0%       |
|          1 |        2181 |         17506 | 11.1%       |
|          2 |        3732 |         15335 | 19.6%       |
|          3 |        2172 |         17084 | 11.3%       |
|          4 |       16098 |           129 | 99.2%       |
|          5 |       18165 |           188 | 99.0%       |
|          6 |        1771 |         14536 | 10.9%       |
+------------+-------------+---------------+-------------+

which suggests that there is at least one missed goniometry datum; testing joint=0

(this is for info not a blocker)

graeme-winter

Firstly thank you for the changes - this has been on my to-do for a very long time, and I appreciate the work which went in to getting this done.

As mentioned in the review comment it feels slow to use but I would be the first to recognise working-but-slow is vastly superior to broken.

I also note that indexing a 7 sweep run did not work correctly, where indexing the sweeps one-at-a-time worked fine, so there is some geometric infelicity in the metadata reading (or writing) which would benefit from some attention in a subsequent PR

I would like this to be merged as is, I appreciate the addition of this capability; with the frame renumbering workaround from @biochem-fan this works nicely and adds real capability to DIALS.

I am still waiitng for the processing of my 7-scan run to finish but I am mid-way through integrating it so I am confident that this is a good change set.

graeme-winter · 2023-07-26T11:09:26Z

src/dxtbx/format/FormatROD.py

+ epoch=None,
+ )
+
+ def get_raw_data(self):


By way of a narrative; spot finding feels slow which I suspect is down to the fact that the get_raw_data() method is itself slower than the equivalent e.g. from CBF or HDF5 data.

I mark this as an opportunity for future improvement perhaps as a C++ implementation rather than a bug here.

biochem-fan · 2023-07-26T11:26:54Z

Thank you very much for testing on another dataset.

I also note that indexing a 7 sweep run did not work correctly, where indexing the sweeps one-at-a-time worked fine, so there is some geometric infelicity in the metadata reading (or writing) which would benefit from some attention in a subsequent PR

Indeed, it currently ignores all axes except the last one so I don't expect joint indexing works.

graeme-winter · 2023-07-26T11:35:01Z

End result:

                                            Suggested   Low    High  Overall
High resolution limit                           1.45    3.93    1.45    1.42
Low resolution limit                           78.78   78.90    1.47   78.78
Completeness                                   89.2   100.0    27.6    85.3
Multiplicity                                   12.8    21.0     1.1    12.8
I/sigma                                        33.2   120.2     1.0    33.1
Rmerge(I)                                     0.044   0.025   0.443   0.044
Rmerge(I+/-)                                  0.041   0.022   0.767   0.041
Rmeas(I)                                      0.045   0.026   0.626   0.045
Rmeas(I+/-)                                   0.043   0.023   1.085   0.043
Rpim(I)                                       0.010   0.005   0.443   0.010
Rpim(I+/-)                                    0.013   0.006   0.767   0.013
CC half                                       1.000   1.000   0.638   1.000
Anomalous completeness                         79.8   100.0     1.6    75.7
Anomalous multiplicity                          7.3    12.7     1.0     7.2
Anomalous correlation                         0.359   0.725   0.000   0.359
Anomalous slope                               0.797                        
dF/F                                          0.051                        
dI/s(dI)                                      0.717                        
Total observations                           242509   25120     317  242598
Total unique                                  18911    1197     288   18999

I think this is most satisfactory

graeme-winter · 2023-07-26T11:36:42Z

Weirdly though:

Grey-Area work :) $ dials.image_viewer integrated.*
DIALS (2018) Acta Cryst. D74, 85-97. https://doi.org/10.1107/S2059798317017235
Traceback (most recent call last):
  File "/Users/graeme/git/dials/build/../modules/dials/src/dials/command_line/image_viewer.py", line 217, in <module>
    run()
  File "/Users/graeme/git/dials/conda_base/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/Users/graeme/git/dials/build/../modules/dials/src/dials/command_line/image_viewer.py", line 193, in run
    params, options = parser.parse_args(args, show_diff_phil=True)
  File "/Users/graeme/git/dials/modules/dials/src/dials/util/options.py", line 876, in parse_args
    params, args = self._phil_parser.parse_args(
  File "/Users/graeme/git/dials/modules/dials/src/dials/util/options.py", line 558, in parse_args
    importer = Importer(
  File "/Users/graeme/git/dials/modules/dials/src/dials/util/options.py", line 207, in __init__
    self.unhandled = self.try_read_experiments_from_images(
  File "/Users/graeme/git/dials/modules/dials/src/dials/util/options.py", line 284, in try_read_experiments_from_images
    experiments = ExperimentListFactory.from_filenames(
  File "/Users/graeme/git/dials/modules/dxtbx/src/dxtbx/model/experiment_list.py", line 533, in from_filenames
    format_class = find_format.find_format(filename)
  File "/Users/graeme/git/dials/modules/dxtbx/src/dxtbx/datablock.py", line 270, in find_format
    self._format_class = get_format_class_for_file(filename)
  File "/Users/graeme/git/dials/modules/dxtbx/src/dxtbx/format/Registry.py", line 121, in get_format_class_for_file
    if scheme in format_class.schemes and format_class.understand(image_file_str):
  File "/Users/graeme/git/dials/modules/dxtbx/src/dxtbx/format/FormatROD.py", line 50, in understand
    hdr = f.read(256).decode("ascii")
UnicodeDecodeError: 'ascii' codec can't decode byte 0x93 in position 0: ordinal not in range(128)

graeme-winter · 2023-07-26T11:39:07Z

@dagewa please could you try

dials.image_viewer indexed.*

or equivalent on any data set

With this change enabled

I think it tries to read *.refl and fails

graeme-winter · 2023-07-26T11:39:46Z

dials.image_viewer integrated.*

etc. now fails on e.g. CBF or HDF5 data

ndevenish · 2023-07-26T11:44:48Z

This looks like it's trying to read the .refl file, which isn't a plaintext. It's correctly failing, but failing to return False; IIRC this was an intentional because the Registry system used to use exceptions as "Format not understood" but this made it impossible to separate out actual errors from failing to read the file.

ndevenish · 2023-07-26T11:47:53Z

This should handle this issue

graeme-winter · 2023-07-26T11:50:12Z

This should handle this issue

It did thank you

dagewa · 2023-07-26T12:23:55Z

This looks like it's trying to read the .refl file, which isn't a plaintext. It's correctly failing, but failing to return False; IIRC this was an intentional because the Registry system used to use exceptions as "Format not understood" but this made it impossible to separate out actual errors from failing to read the file.

Ah yes, good point. Thanks!

Partial support for Rigaku Oxford Diffraction format images. --------- Co-authored-by: Takanori Nakane <nakane.t@gmail.com> Co-authored-by: Nicholas Devenish <ndevenish@gmail.com>

dagewa added 3 commits July 20, 2023 09:33

Copy FormatROD.py from @biochem-fan's gist as-is. Source:

3ce1d3e

https://gist.githubusercontent.com/biochem-fan/5f76423120e92828f03a4763a622a1e9/raw/a16a5c223a2bb7ed4764dd720d26c6f55aef5b91/FormatROD.py

Satiate flake8, black et al. Also, more efficient understand method

64cdd36

News

3103b32

dagewa requested review from graeme-winter and biochem-fan July 20, 2023 08:55

Rename newsfragments/xxx.feature to newsfragments/645.feature

1bd01c8

bugfix

cf07f9f

biochem-fan approved these changes Jul 20, 2023

View reviewed changes

dagewa mentioned this pull request Jul 20, 2023

Handling image templates that don't use zero-padding #646

Closed

Add note about Rigaku non-zero-padded image templates, which cause

6e5e303

problems with dials.import.

graeme-winter approved these changes Jul 26, 2023

View reviewed changes

Handle failure when decoding incompatible binary files

3ed0b46

dagewa merged commit 180fac8 into cctbx:main Jul 26, 2023
11 checks passed

biochem-fan mentioned this pull request Aug 2, 2023

FormatROD - multi-axis goniometer and faster decompression #186 #653

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FormatROD #645

FormatROD #645

dagewa commented Jul 20, 2023

biochem-fan commented Jul 20, 2023 •

edited

Loading

dagewa commented Jul 20, 2023

codecov bot commented Jul 20, 2023 •

edited

Loading

dagewa commented Jul 20, 2023

dagewa commented Jul 20, 2023

biochem-fan left a comment

dagewa commented Jul 20, 2023

graeme-winter commented Jul 26, 2023

graeme-winter commented Jul 26, 2023

graeme-winter commented Jul 26, 2023

graeme-winter left a comment

graeme-winter Jul 26, 2023

biochem-fan commented Jul 26, 2023

graeme-winter commented Jul 26, 2023

graeme-winter commented Jul 26, 2023

graeme-winter commented Jul 26, 2023

graeme-winter commented Jul 26, 2023

ndevenish commented Jul 26, 2023

ndevenish commented Jul 26, 2023

graeme-winter commented Jul 26, 2023

dagewa commented Jul 26, 2023

FormatROD #645

FormatROD #645

Conversation

dagewa commented Jul 20, 2023

biochem-fan commented Jul 20, 2023 • edited Loading

dagewa commented Jul 20, 2023

codecov bot commented Jul 20, 2023 • edited Loading

Codecov Report

dagewa commented Jul 20, 2023

dagewa commented Jul 20, 2023

biochem-fan left a comment

Choose a reason for hiding this comment

dagewa commented Jul 20, 2023

graeme-winter commented Jul 26, 2023

graeme-winter commented Jul 26, 2023

graeme-winter commented Jul 26, 2023

graeme-winter left a comment

Choose a reason for hiding this comment

graeme-winter Jul 26, 2023

Choose a reason for hiding this comment

biochem-fan commented Jul 26, 2023

graeme-winter commented Jul 26, 2023

graeme-winter commented Jul 26, 2023

graeme-winter commented Jul 26, 2023

graeme-winter commented Jul 26, 2023

ndevenish commented Jul 26, 2023

ndevenish commented Jul 26, 2023

graeme-winter commented Jul 26, 2023

dagewa commented Jul 26, 2023

biochem-fan commented Jul 20, 2023 •

edited

Loading

codecov bot commented Jul 20, 2023 •

edited

Loading