FormatMRC invert slow axis for non-FEI files

[dagewa]

As pointed out by @huwjenkins, there is a lack of consensus regarding the position of the origin for images represented by MRC files. Software like IMOD and EMAN2 take the origin at bottom left, whereas ThermoFisher adopts the typical raster-scan convention of having the origin at the upper-left.

I think this is a symptom of the abuse of MRC (a map data format) to hold image data.

Anyway, we suspect that any data from a non-FEI source (i.e. lacking the FEI extended header) will use the bottom-left convention. This change ensures that by inverting the direction of the slow axis. Then also the distance must be inverted. Note, the actual image array is left unchanged.

A minor annoyance for non-FEI images is that if you want to override the distance at import you will have to provide a negative distance.

[codecov]

Codecov Report

Attention: Patch coverage is 25.00000% with 3 lines in your changes are missing coverage. Please review.

Project coverage is 41.72%. Comparing base (2732092) to head (12524e6).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #736      +/-   ##
==========================================
- Coverage   41.73%   41.72%   -0.01%     
==========================================
  Files         187      187              
  Lines       16857    16861       +4     
  Branches     3226     3227       +1     
==========================================
+ Hits         7035     7036       +1     
- Misses       9173     9175       +2     
- Partials      649      650       +1     

[biochem-fan]

I am afraid that this change is too disruptive; scripts and data processing guides to process a non-FEI data written for older versions of DIALS will break.

It does not really matter how each image viewer displays an image on a screen. What matters is the physical model. RELION's image display is indeed flipped compared to IMOD and EMAN2, but coordinates and Euler angles written in a STAR file are consistent with the standard convention.

I would keep the slow axis and the sign of the detector distance as they are and instead absorb the difference in the goniometer axis (if necessary). People have to find and set the right goniometer axis for every new scope anyway.

we suspect that any data from a non-FEI source (i.e. lacking the FEI extended header) will use the bottom-left convention.

I am not so sure. Moreover, physical "bottom-left" of a camera does not necessarily match "bottom-left" of a sample, because the projection lens system might rotate the image. For some strange reasons, in some scopes, images can be further rotated or even flipped before written. This is why gain references must be sometimes rotated or flipped before applied to raw movies.

Because one has to painstakingly figure out the coordinate system for every scope anyway, I hope you don't change it further. One has to figure out the new DIALS system again.

[dagewa]

I agree that the image display doesn't matter, and it is the physical model that matters. Prior to this change, the physical model is wrong for e.g. the CAU-36 example described in https://doi.org/10.1016/j.str.2023.07.004 This leads to inversion of Friedel indices. With this change, such data sets (anything converted to MRC by RED or instamatic?) will be correctly interpreted by DIALS.

Note that you cannot fix this by only changing the goniometer axis - the example shows that the axis is flipped and inverted, but still the Miller indices are inverted.

I think your point that we cannot trust that all non-FEI MRCs will use a bottom left convention is crucial though. If it really is "Wild West" out there like it sounds then perhaps there is no sense in trying to guess the right convention.

[dagewa]

And of course your point that it would break existing scripts and guides is valid. I do not know of so many, but experience does tell me that you tend to find out about these things only after you break something.

[huwjenkins]

For some strange reasons, in some scopes, images can be further rotated or even flipped before written.

Isn't this so images from different cameras mounted on the same microscope show the sample in the same orientation? Certainly on our Glacios, where the Falcon4 and Ceta-D are inserted from opposite sides, in the camera control software one detector is set to have the image rotated by 90° and the other by 270°.

[biochem-fan]

Note that you cannot fix this by only changing the goniometer axis - the example shows that the axis is flipped and inverted, but still the Miller indices are inverted.

Yes, this matters if you perform dynamical refinement to determine absolute hands. For kinematical refinement, not.

Your paper says:

This is particularly dangerous for electron diffraction, where Friedel’s law is broken to some degree by dynamic diffraction effects. Indeed, this effect can be exploited to determine the absolute structure of chiral molecules

This is true, but in practice, the diffraction geometry is deduced by checking the hand of a known standard first. This geometry is specified when importing image of unknown samples. If you are lucky, the scope administrator sets this to the data acquisition software so that it flips the image array before writing MRC movies, but this is rare.

Not for MicroED but for SPA, we wrote the following in the RELION 3 paper:

Because an unknown number of mirroring operations may have been introduced during the data acquisition and image processing, the actual direction of curvature of the Ewald sphere is often unknown for a given data set.

Also note that the direction can be different between different camera lengths of a single scope, because some intermediate lenses may be switched on or off, leading to different numbers of image flips.

I think your point that we cannot trust that all non-FEI MRCs will use a bottom left convention is crucial though. If it really is "Wild West" out there like it sounds then perhaps there is no sense in trying to guess the right convention.

I am afraid it is.

I do not know of so many, but experience does tell me that you tend to find out about these things only after you break something.

Many people have in-house data processing scripts but do not necessarily publish them. They will be unhappy if something used to work break suddenly and have to figure out the geometry again.

P.S.

anything converted to MRC by RED or instamatic?

I don't understand why so many MicroED people convert images to MRC (or even SMV). Both do not support transparent compression. I always convert to TIFF. But again, some converters flip the slow axis when going from MRC to TIFF, while others not, and the confusion continues.

[biochem-fan]

Isn't this so images from different cameras mounted on the same microscope show the sample in the same orientation?

Possibly.

But the problem is, you can never trust this. EPU, SerialEM, Digital Micrograph etc can have different settings and write movies differently.

[dagewa]

Yes, this matters if you perform dynamical refinement to determine absolute hands. For kinematical refinement, not.

Indeed this concerned me greatly for the hemozoin project, where handedness was a key result. In that case we used a dxtbx-based script to convert images to TIFF for use by PETS. In a few other cases DIALS has now been used to prepare data directly for dynamic-diffraction refinement in Jana2020 (it works, but differences are not as strong as from PETS data)

[dagewa]

I don't understand why so many MicroED people convert images to MRC (or even SMV).

Hear hear!

[huwjenkins]

I think your point that we cannot trust that all non-FEI MRCs will use a bottom left convention is crucial though. If it really is "Wild West" out there like it sounds then perhaps there is no sense in trying to guess the right convention.

It does sound like this is the case.

But again, some converters flip the slow axis when going from MRC to TIFF, while others not, and the confusion continues.

OK. I'll change the wiki page into a general warning about slow axis flip when converting to/from MRC and that can serve as a reference to back up your comment ⬇️

I don't understand why so many MicroED people convert images to MRC (or even SMV).

me neither.

[dagewa]

Ok, and I'll close the PR. It is true I am fed up dealing with deficient data formats.