CHANGELOG.rst

DIALS 3.10.0 (2022-06-09)

Features

• The DIALS code package now uses src/ layout. You will need to at least libtbx.refresh if on a development install. (#2077)
• dials.cosym: Enable skipping of unit cell clustering by setting unit_cell_clustering.threshold parameter to 0 or None. (#2058)
• dials.export: Add extra unmerged data categories for mmcif output, conforming to the latest mmcif_pdbx.dic. Make v5 the default (rather than v5_next). (#2078)
• dials.export: MTZ files are now exported with geometry in the Cambridge frame. (#2054)
• dials.index: Performance improvements for serial indexing (dials.stills-process, dev.dials.ssx_index). (#2035)
• dials.merge: New implementation of the French & Wilson (1978) algorithm for correction of negative intensities when estimating amplitudes. This implementation makes use of the standardized median as an M-estimator for the average intensity of resolution bins, which makes the procedure robust against the presence of very negative intensities. (#2100)
• dials.powder_calibrate: Add detector distance calibration. (#2075)
• dials.refine: New separate_images option performs outlier rejection on each image independently. (#2036)
• dials.refine: Parallelise outlier rejection to reduce overall run times. (#1427)
• dials.refine: Use sparse storage for scan-varying runs, reducing memory requirements and run times. (#2022)
• dials.scale: Allow use of a pdb model to calculate target intensities for scaling (phil option target_model). (#2053)
• dials.stills_process: Validate command-line arguments to prevent confusion when there is a typo. (#2106)
• dev.dials.ssx_index, dev.dials.ssx_integrate: Add option output.nuggets=, which can be used to specify a directory to which in-process results are stored. (#2114)
• Add a CMake build of DIALS. (#2096)

Bugfixes

• dials.cluster_unit_cell: Modify test so that it runs on Windows. (#2027)
• dials.cluster_unit_cells: Correctly handle cases with only one input crystal. (#2120)
• dials.combine_experiments: Exit with a helpful error message, if experiments have the same identifiers. (#2069)
• dials.export: Fix crash for exporting ssx data. (#2126)
• dials.find_bad_pixels: Output a mask file, as expected in the phil scope. Remove unimplemented png output option. (#2122)
• dials.image_viewer: Fix downstream SEGV involving wxPython. (#2134)
• dials.image_viewer: Fixed user setting for show_beam_center= being overridden. (#2103)
• dials.image_viewer: the n_iqr value for radial_profile thresholding is now correctly handled. (#2116)
• dials.import: When trying to import a missing file, say which file was missing. Print a warning if trying to import with a wildcard and no files were found. (#1863)
• dials.refine_bravais_settings: Ensure that the reported reindexing operators correctly map the input symmetry to the given Bravais settings, regardless of whether the input symmetry was a primitive or non-primitive setting. (#2105)
• dials.reindex: Fail with a helpful error message when attempting to reindex to a left-handed cell. (#1779)
• dials.scale: If scaling against a target, do targeted outlier rejection. (#2052)
• dials.scale: Respect user supplied high resolution limit in summary table. (#2118)
• dials.show: Avoid crash when an experiment does not have an imageset. (#2056)
• dev.dials.ssx_index: Fix reporting of results for h5 files, skip indexing of an image if fewer than min_spots= strong spots. (default 10). (#2055)
• dev.dials.ssx_index: Handle case where the input strong.refl file has no spots for some images. (#2039)
• dev.dials.ssx_integrate: Correctly handle input data containing multiple imagesets. (#2124)
• Handle reflection simulation case where test reflections could fail to generate. (#2094)

Improved Documentation

• dials.model_background: Add help message and test program. (#1109)
• Remove reference to nproc in tutorials where it is not needed. (#2030)
• Update DPF3 part 2 tutorial, to match current output. (#2030)
• Add dials.two_theta_refine to documentation (#2061)

Deprecations and Removals

• dials.find_shared_models has been retired. The command will now redirect users to dials.show, in combination with the show_shared_models=True option. This command stub will be removed in a future version. (#1070)
• dials.integrate: The unused background.algorithm=median has been removed. (#2066)
• The API interface dials.command_line.dials_import.ImageImporter is now deprecated. Please use ...dials_import.do_import instead. (#2080)

Misc

• #1973, #2037, #2038, #2041, #2043, #2047, #2051, #2062, #2065, #2070, #2071, #2073, #2074, #2079, #2081, #2082, #2083, #2086, #2087, #2088, #2089, #2091, #2092, #2093, #2095, #2099, #2101, #2102, #2104, #2112, #2119, #2131, #2133

DIALS 3.8.6 (2022-06-07)

• Resolve xia2 installation issues for downstream packaging.

DIALS 3.8.5 (2022-06-01)

Features

• dials.stills_process: validate command-line arguments to prevent confusion when there is a typo (#2106)

Bugfixes

• dials.show: Fix display of unknown vector columns. (#2048)
• dials.image_viewer: Fixed user setting for show_beam_center= being overridden. (#2103)

DIALS 3.9.2 (2022-05-09)

Bugfixes

• dials.show: Fix display of unknown vector columns. (#2048)

DIALS 3.8.4 (2022-04-01)

Bugfixes

• dials.scale: Fix crash when a dataset is filtered out during the scaling process (issue #2045). (#2045)

DIALS 3.9.1 (2022-03-31)

Bugfixes

• dials.scale: Fix crash when a dataset is filtered out during the scaling process (issue #2045). (#2045)

DIALS 3.9.0 (2022-03-14)

Features

• Bootstrap now allows creating a Python 3.10 environment. This should be considered experimental at this stage, and may fail because not all our dependencies have Python 3.10 support yet. (#1866)
• dials.export: Add SHELX .hkl file output. (#1925)
• dials.background now writes to a log file. (#1948)
• dials.cluster_unit_cell: Add option output.clusters=True/False to generate output files for each cluster generated by splitting the dendrogram at the given threshold. (#1950)
• Add reflection_table.match_by_hkle method, to match reflections between tables that have the same miller index and entering flags. (#1951)
• Add experimental dev.dials.ssx_integrate script for profile modelling and integration of SSX data, including ellipsoid profile modelling. (#1974)
• dials.reindex: Allow reindexing using multi-crystal reference data files. (#1977)
• add flex.reflection_table.concat method, to concatenate a list of reflection tables, including handling their experiment identifiers and ids (#1994)
• For data reduction programs, allow exclude_images option to take a single multi-sweep command, e.g. exclude_images=0:100:120,1:150:180 (#1996)
• Enable spot-finding threshold algorithms to use information about experimental models. (#2001)
• dials.find_spots: Added new spotfinder.threshold=radial_profile threshold algorithm. This calculates an average background in 2θ shells, and identifies peak pixels at a user-controllable level above the background. This simple method is particularly appropriate for cases with strong rotationally-symmetric background, such as electron diffraction images. An optional blurring function helps to suppress noise peaks and to join split spots. (#2009)
• dials.export: Add PETS 2 exporting with format=pets. This is used for processing electron diffraction data. (#2014)
• New tool - dials.powder_calibrate to help calibrate the geometry of an electron powder pattern. (#2016)
• Colours in plots: Some of our plots and figures were still using the Matplotlib colour map Jet. Matplotlib moved some time ago to using the perceptually uniform colour map Viridis, which was designed to be relatively colourblind-friendly, as its default. In this version of DIALS, we too have moved to use Viridis for our Plotly plots. With thanks to Ammaar Saeed (ammsa23) for this change. (#2026)

Bugfixes

• dials.refine could in some rare cases introducing an unphysical beam polarization vector. (#1939)
• dials.scale: Fixes to properly handle partiality of ssx data (#1965)
• This round includes modifications to enable Kapton absorption correction for higher angles of rotation of the Kapton tape. (#1968)
• ThreadPool and Socket resources were not properly closed in dials.find_spots_client. (#1976)
• Fix regression in installer python 2 compatibility. (#1990)
• Avoid using the uuid module, to avoid mpi errors on Python 3.8. (#2000)
• dials.refine: Avoid spike in memory usage while saving reflections. (#2024)

Improved Documentation

• Improved the "small molecule" tutorial by including the symmetry determination, scaling and export to e.g. SHELX format output. (#1900)
• Add a new multi-crystal tutorial, discussing the analysis of Br-lysozyme microcrystal data with dials.cosym and xia2.multiplex. (#1960)
• Update the betalactamase tutorial to use automatic scan-varying refinement. (#1971)
• Remove outdated developer tutorial. (#2003)
• Documentation font has changed to sans-serif. (#2010)

Deprecations and Removals

• Starting with this release DIALS requires a minimum Python version of 3.8. Bootstrap no longer allows the creation of Python 3.7 environments. (#1866)

Misc

• #1576, #1930, #1953, #1966, #1967, #1972, #1975, #1978, #1986, #1989, #1995, #1999, #2013, #2015, #2017, #2018, #2019, #2020, #2021, #2023

DIALS 3.8.3 (2022-02-22)

Bugfixes

• dials.cosym: Fix crash for edge case of a full dataset being excluded by the resolution filter (#1993)
• dials.two_theta_refine: Fix crash if running on scaled data with exclude_datasets=. (#2006)
• Fix downloads failing on MacOS with "426 Upgrade Required" (#2012)

DIALS 3.8.2 (2022-02-07)

No significant changes.

DIALS 3.8.1 (2022-01-25)

Features

• It is now possible to explicitly bootstrap all supported python versions. (#1988)

Bugfixes

• dials.refine: Fix crash from recording of parameter correlations in history.json. (#1923)
• Correct reporting of phi angle in scan-varying model plots. (#1929)
• dials.sequence_to_stills: correct error in the crystal model for scans that do not start from image 1. (#1933)
• dials.show: only show format class if meaningful (i.e. not Format or FormatMultiImage) (#1981)

Improved Documentation

• Update some Cosym and resolution-related PHIL descriptions. (#1969)

DIALS 3.8.0 (2022-01-11)

Features

• dials.indexed_as_integrated: manipulate an indexed reflection file to look as if it were summation integrated. This simply takes the spot intensities that have been indexed, assigns a resolution and sets the summation integrated flag. Allows symmetry analysis using intensities and scaling on indexed data for very rapid feedback data processing. (#1912)
• All command line programs now allow passing -h argument multiple times to increase verbosity. (#1920)
• Add Dockerfile to automatically build and push images on new releases. (#1936)
• Add experimental dev.dials.ssx_index script to index a block of ssx data. (#1955)
• Add alias dials.rlv for dials.reciprocal_lattice_viewer, and dials.rbs for dials.refine_bravais_settings. (#1959)

Bugfixes

• dials.import: Support arbitrary P1 known unit cells. (#1880)
• Utility fixes: Better handle cases of missing partiality data in reflection files. Treat missing resolution values as 0.0 instead of being empty. (#1911)
• dials.scale: Add missing "expids_and_image_ranges" information to the json output. This is required for some of the plots in dials.report output. (#1913)
• Copy docker entrypoint script with exec permissions. (#1940)
• Don't line-wrap command-line help messages (#1954)
• dials.check_indexing_symmetry: correctly handle d_max parameter if left at default value when d_min set. (#1957)
• Fixed bug that prevented the ability to plot absorption end of max and min due to Kapton (#1962)

Improved Documentation

• Updated MyD88 tutorial to make use of the new dials.find_rotation_axis command. (#1885)
• Document the dxtbx convention for representing the goniostat rotation operator \mathbf{R} on the conventions page. of the online documentation. (#1917)
• Update tutorial for DUI 2021.11.1. (#1938)

Deprecations and Removals

• The cosym nproc= and dials.util.parallel_map warnings have been made more visible. (#1909)

Misc

• #1907, #1908, #1910, #1928, #1947

DIALS 3.7.2 (2021-12-02)

Features

• dials.integrate: When determining available memory, take into account MemoryProvisioned from HTCondor machine ad if the _CONDOR_JOB_AD environment variable is set. nproc=auto: Take into account CpusProvisioned from HTCondor machine ad. (#1943)

Bugfixes

• Read _CONDOR_JOB_AD not _CONDOR_MACHINE_AD (#1945)

DIALS 3.7.1 (2021-11-17)

Bugfixes

• dials.export: No longer allow (erroneous) MTZ export for multiple experiments with multiple space groups. (#1915)
• dials.export: No longer fails for XDS_ASCII and SADABS export with intensity=auto. (#1926)
• dials.report: Fix broken json output option. Include more graphs in json output. (#1932)

DIALS 3.7.0 (2021-11-01)

Features

• Bootstrap support for MacOS M1 platforms. (#1841)
• New dials.find_rotation_axis program optimises the rotation axis from strong spot positions prior to indexing. (#1884)
• dials.import: Allow importing templates with no template characters. (#1840)
• dials.stills_process: Performance improvements in Kapton absorption correction and in rare cases of highly mosaic crystals. (#1846)
• dials.image_viewer Coordinates are now given in fast, slow order. (#1849)
• dials.image_viewer: Crystal basis vectors are now shown in the same colour as their predictions. (#1855)
• dials.image_viewer: Add the option to display the rotation axis (#1856)
• dials.image_viewer: Draw resolution rings for curved detectors. (#1899)
• dials.import: Unhandled files are now by default ignored. This means that e.g. *.log files alongside images will no longer prevent a successful import. Set ignore_unhandled=False to restore the previous behaviour. (#1881)
• dials.scale: Allow fixing of a particular correction with e.g. physical.correction.fix=absorption. (#1883)
• Installer now accepts a --raw-prefix option to use the target destination directly, instead of in a dials-X.Y subdirectory. (#1896)

Bugfixes

• dials.compute_delta_cchalf: Fix crash when only using passing dataset/group. (#1892)
• dials.find_bad_pixels: Pixel coordinates are now reported in row-major order, and mask value is now set to 16, which corresponds internally to "noisy pixel". (#1876)
• dials.find_rotation_axis: removed unused parameter optimise={True|False}. (#1898)
• dials.report: Don't show otherwise empty sections. (#1875)

Improved Documentation

• Improvements to 3DED tutorials. (#1850)
• SARS-CoV-2 main protease tutorial: process in C2 rather than I2 setting for consistency with published structures. (#1854)
• Removed outdated lysozyme nanocrystals tutorial. (#1877)
• Add an associated projects page to the website. (#1893)

Deprecations and Removals

• Bootstrap no longer allows creating Python 3.6 environments. (#1852)
• dials.util.mp: deprecate parallel_map() function and remove previously deprecated preserve_exception_message= parameters. (#1860)

Misc

• #1851, #1853, #1862, #1865, #1867, #1869, #1882, #1887, #1888, #1889, #1891, #1894, #1902

DIALS 3.6.2 (2021-09-21)

Bugfixes

• dials.reciprocal_lattice_viewer: In cases with multiple lattices, "Crystal Frame" now aligns all crystal frames, rather than just the first. Unindexed reflections are no longer shown in this mode. (#1868)

DIALS 3.6.1 (2021-09-06)

No significant changes.

DIALS 3.6.0 (2021-08-16)

This is the last release to support Python 3.6. Future releases will require a minimum of Python 3.7.

Features

• DIALS bootstrap now creates a Python 3.9 environment by default (#1735)
• New program: dials.reference_profile_viewer for viewing reference profiles dumped by dials.integrate when using the debug.reference.output=True option. (#1759)
• dials.combine_experiments: Unindexed reflections are now included in the combined output (#1760)
• dials.image_viewer: Image overlays are now accumulated over stacks of images (#1750)
• dials.image_viewer: Allow control of the basis vector scale from the settings window (#1780)
• dials.image_viewer: Better colour choice for text overlays. Labels will now be light grey on Black, or Dark grey on White. The previous settings were sometimes hard to read on narrow-contrast images. (#1781)
• dials.merge: Include DANO/SIGDANO columns in output .mtz when anomalous=True and truncate=True (#1809)
• dials.reciprocal_lattice_viewer: Show resolution on the "nearest point" label (#1770)
• dials.reciprocal_lattice_viewer now shows the path to the reflections in the title bar (#1771)
• dials.reciprocal_lattice_viewer: The default marker size now scaled automatically based on the data density (#1773)
• dials.scale: Always enable absorption correction if the absorption_level= parameter is set. Previously it was only enabled for sweeps >= 60° or if absorption_correction=True. (#1793)
• dials.scale: Allow a shared absorption correction between sweeps if using the physical model, with the option share.absorption=True. Extra absorption correction plots have also been added; and multiple sweeps are now aligned to the same reference frame. (#1811)
• API: ...scaling_library.scaled_data_as_miller_array now sets wavelength in the returned miller.array (#1808)
• reflection_table.match() now returns flex.size_t index arrays, instead of flex.int. (#1784)
• New bootstrap option: --conda to install with miniconda instead of micromamba. (#1730)

Bugfixes

• dials.combine_experiments: Correctly preserve mapping to images. This affects dials.image_viewer and dial.reciprocal_lattice_viewer. (#1093)
• dials.compute_delta_cchalf: Unwarranted precision in the output has been reduced (#1751)
• dials.find_spots: Fix counting of imagesets in histogram output (#1827)
• dials.image_viewer: Add buttons to clear unit cell and generic ring display (#1777)
• dials.image_viewer: Fix various minor behavioural bugs in the spot-finding and image type controls. (#1796)
• dials.import: Fail gracefully when # is missing from template. (#1840)
• dials.integrate: change default filename of debug reference profile to reference_profiles.pickle. (#1747)
• dials.integrate: Change default configuration so that unintegrated reflections are not retained. This helps reduce memory usage of downstream tools. Set output_unintegrated_reflections=True to restore the previous behaviour. (#1753)
• dials.integrate: ensure imageset_ids are always output. Affected use of image viewer, reciprocal lattice viewer on multi-sweep data. (#1762)
• dials.reciprocal_lattice_viewer: When starting with black_background=False, ensure the rotation axis and beam vector are displayed in black. (#1540)
• dials.reciprocal_lattice_viewer: More robust beam centre control that works for multiple panel detectors (#1842)
• dials.refine_bravais_settings: correctly report mI Bravais settings (#1825)
• dials.split_experiments: Update the imageset_id column in the output reflection files. (#1792)
• Don't fail bootstrap.py if a submodule is missing a reference (#1834)
• Correctly handle reflection imageset_id column in dials.scale, dials.cosym, and dials.symmetry. (#1763)

Improved Documentation

• dials.anvil_correction: Made a small improvement to the developer documentation. (#1788)
• Fix help string for best_monoclinic_beta= parameter (for dials.cosym, dials.refine_bravais_settings and dials.symmetry) (#1833)
• Added a new tutorial on 3DED/MicroED data processing. (#1837)
• Add a "Getting started" page to the documentation on the website. (#1844)
• Add a tutorial on processing small molecule 3DED data. (#1847)

Deprecations and Removals

• The previously deprecated dials.resolutionizer command has been removed. Please use dials.estimate_resolution instead. (#1330)
• The previously deprecated dials.refine parameter trim_scan_edges has been removed. Please use scan_margin=... instead. (#1374)
• The previously deprecated Spotfinder()() interface has been removed. Please use Spotfinder().find_spots() instead. (#1484)
• The previously deprecated dials.util.masking.MaskGenerator has been removed. Please use dials.util.masking.generate_mask instead. (#1569)
• The bootstrap option --mamba has become the default and will be removed in the future. (#1730)
• dials.anvil_correction: Drop compatibility support for SciPy < 1.4 (#1787)

Misc

• #1746, #1733, #1752, #1755, #1756, #1764, #1767, #1772, #1783, #1789, #1791, #1794, #1795, #1799, #1802, #1804, #1806, #1807, #1812, #1816, #1817, #1823, #1830, #1835, #1836, #1839

DIALS 3.5.4 (2021-07-27)

Bugfixes

• dials.stills_process: Fix case where imagesets and experiment filenames could potentially disagree (#1814)
• dials.scale: Fix incorrect output files, for targeted scaling with more than one target dataset. (#1815)
• dials.image_viewer: Fix opening datasets with load_models=False (#1818)

DIALS 3.5.3 (2021-07-12)

Bugfixes

• dials.image_viewer: Fix the basis_vector_scale= parameter. (#1769)

DIALS 3.5.2 (2021-06-28)

Bugfixes

• dials.image_viewer: Fix display of spotfinding intermediates (threshold, dispersion, etc) when viewing multiple still experiments (#1734)
• dials.image_viewer: Stacking images no longer gives incorrect results for multi-sweep data beyond the first sweep (#1758)

DIALS 3.5.1 (2021-06-14)

No significant changes.

DIALS 3.5.0 (2021-05-27)

Features

• dials.integrate: Avoid crash when data is too large to process in memory, by splitting into subsets (#1392)
• New bootstrap options: --mamba to install with micromamba, and --clean to remove installation caches immediately after completion. (#1676)
• dials.find_spots_server: Faster filtering of reflections by resolution (#1680)
• dials.scale: Add option error_model.grouping= to control refinement of either individual or grouped error models during scaling (#1684)
• dials.scale: Added physical.absorption_level=[low|medium|high] option for automatic setting of suitable absorption correction parameters. (#1688)
• dials.cosym: Significantly faster calculation of Rij matrix of pairwise correlation coefficients (#1693)
• dials.sort_reflections and dials.merge_reflection_lists are now available without a dev. prefix. (#1703)
• New command: dials.find_bad_pixels to identify pixels which are identified as signal in >= 50% of images (#1710)
• dials.image_viewer: Add selector to choose between a new default "image" and traditional "lab" coordinate frames. "image" frame attempts to align the fast/slow axes of the detector panels to screen x and y coordinates, so overall detector rotations will mostly be invisible. "lab" frame is the previous projection, where rotated detectors will appear rotated. dials.export_bitmaps: Gained this same projection= option. (#1716)
• dials.find_spots and dials.integrate: nproc= now works with N > 1 on Windows. (#1724)

Bugfixes

• Fix rare crash in symmetry calculations when no resolution limit could be calculated (#1641)
• dials.report: Add units of pixels / images to centroid difference histograms (#1677)
• dials.refine: Scan-varying refinement failed when trim_scan_to_observations=False was used. (#1686)
• dials.spot_counts_per_image: Show an explicit error if given data that isn't spotfinding output (i.e. unindexed reflections/experiments). (#1690)
• dials.integrate: Improved background model variance calculation for integrating detectors. (#1692)
• dials.stills_process: improve processing performance by preventing re-reading of image data (#1705)
• dials.background: Correctly identify signal pixels for integrating detectors, and respect pre-calculated masks. (#1726)
• dials.integrate: Fixed bug in memory-use calculation for multi-sweep integration runs (#1728)

Improved Documentation

• Remove remaining 'master' references in the documentation. (#1632)

Deprecations and Removals

• The previously deprecated dials.util.masking.MaskGenerator now prints a user warning. Please use dials.util.masking.generate_mask instead. (#1643)
• dials.cosym: Remove clustering code as this is no longer a necessary part of determination of symmetry or reindexing operations, and serves no useful purporse. (#1647)
• dials.cosym: nproc= parameter is deprecated. The algorithm is much faster on single cores. (#1693)
• The pytest option --runslow was retired. The tests that it triggered will now always run. (#1695)
• dev.dials.csv has been deprecated. Similar functionality is available with dials.export format=json. (#1708)
• dials.util.mp: The preserve_exception_message argument has been deprecated. (#1722)

Misc

• #1631, #1633, #1648, #1649, #1652, #1661, #1672, #1673, #1674, #1675, #1676, #1678, #1679, #1687, #1696, #1697, #1698, #1701, #1706, #1707, #1711, #1713, #1717, #1718, #1720

DIALS 3.4.3 (2021-04-20)

Bugfixes

• dials.scale: Fix crash when full-matrix minimisation is unsuccessful due to indeterminate normal equations. (#1653)
• dials.scale: Fix crash when no reflections remain after initial filtering. (#1654)
• dials.export: Fix error observed with format=mmcif for narrow sweeps with low symmetry (#1656)
• Fix image numbering inconsistency in ascii histogram of per-image spot counts (#1660)
• dials.find_spots_server: Significant performance improvement for HDF5 grid scans. (#1665)

DIALS 3.4.2 (2021-04-12)

Bugfixes

• Log messages from spot finding and integration no longer ignore logging level when using nproc > 1. This mainly affects usage of dials from outside contexts. (#1645)

DIALS 3.4.1 (2021-04-01)

Features

• dials.cosym: Significantly faster via improved computation of functional, gradients and curvatures (#1639)
• dials.integrate: Added parameter valid_foreground_threshold=, to require a minimum fraction of valid pixels before profile fitting is attempted (#1640)

Bugfixes

• dials.cosym: Cache cases where Rij is undefined, rather than recalculating each time. This can have significant performance benefits when handling large numbers of sparse data sets. (#1634)
• dials.cosym: Fix factor of 2 error when calculating target weights (#1635)
• dials.cosym: Fix broken engine=scipy option (#1636)
• dials.integrate: Reject reflections with a high number of invalid pixels, which were being integrated since 3.4.0. This restores better merging statistics, and prevents many reflections being incorrect profiled as zero-intensity. (#1640)

DIALS 3.4.0 (2021-03-15)

Features

• dials.integrate: Profile-fitting improvements; Profile fitting will now be attempted on reflections with masked pixels, and the number of reflections qualifying for profile-fitting on multi-panel detectors has dramatically increased. (#1297)
• dials.import: When using reference_models=, individual components of the model can be excluded with use_beam_reference=, use_gonio_reference= and use_detector_reference=. (#1371)
• flex.reflection_table.match can now match reflections with configurable distance and scaling between any 3-vector column in the reflection tables. The default is still "xyzobs.px.value". (#1398)
• dials.background: Add option output.plot= to save an image to disk, instead of displaying interactively. Image files can now also be used directly. (#1537)
• dials.import: The default tolerance.scan.oscillation= is increased to 3% of the image width, in order to accommodate electron diffraction datasets with poor rotation stages. (#1543)
• dials.background: Add support for multiple imagesets (#1554)
• dials.estimate_resolution: reject Wilson outliers to minimise effect of spurious observations from e.g. ice rings on the resulting resolution estimates (#1580)
• dials.cosym: Use numpy in place of flex for large parts of cosym analysis (#1581)
• dials.cosym: Add option to use scipy L-BFGS-B <https://docs.scipy.org/doc/scipy/reference/optimize.minimize-lbfgsb.html> minimization engine (minimization.engine=scipy) (#1581)
• New masking parameter disable_parallax_correction=False. Set to True to speed up generation of resolution masks by disabling parallax correction (this is only likely to have significant effect when spotfinding is spread across many independent processes). (#1590)
• dials.image_viewer: New parameter basis_vector_scale= to adjust the length of the basis vector overlay (#1598)
• dials.merge: add option to set wavelength_tolerance for MAD datasets (#1609)
• dials.reciprocal_lattice_viewer: Added an option to label the reciprocal lattice point nearest the centre (#1614)
• dials.scale: An additional outlier rejection based on normalised intensities has been added (#1627)

Bugfixes

• dials.image_viewer: Fix various display issues relating to viewing still images (#1463)
• dials.background: Fix crash when writing output plot with bad display configuration (#1550)
• dials.scale: Fix issue of error model not always being carried through after the profile/summation intensity combination step. (#1566)
• Fail bootstrap step if the git checkout fails in a non-interactive or non-posix environment (#1572)
• Fixes working towards direct support of Windows builds:
  • Fix build errors by ensuring conda environment is correctly set up. (#1575)
  • Fix importing using paths with wildcards (#1583)
  • Fix dials.* commands crashing when unicode output is directed to a file (#1602)
  • Fix some type-related test failures (#1608)

Improved Documentation

• Describe how to fix gltbx build failures for development installations on non-RHEL distributions (#1561)
• Replace references to .pickle with reflections / .refl in docstrings (#1619)
• Add documentation for dials.filter_reflections to the website. (#1625)

Deprecations and Removals

• Remove previously deprecated use_trusted_range= parameter from masking configuration (#1156)
• The main development branch of dials was renamed from 'master' to 'main'. (#1546)
• dials.background: The plot= parameter to interactively display the background plot has been removed. Use output.plot= to save to file instead. (#1554)
• Remove *.o files from the DIALS installer package (#1564)
• dials.util.masking.MaskGenerator is deprecated in favour of dials.util.masking.generate_mask (#1569)

Misc

• #1530, #1531, #1532, #1534, #1535, #1536, #1542, #1567, #1570, #1571, #1588, #1593, #1597, #1599, #1600, #1601, #1603, #1604, #1613, #1620, #1621, #1624, #1626, #1630

DIALS 3.3.4 (2021-03-05)

Bugfixes

• dials.import: Selecting a subset of images with image_range= now works on stills (#1592)
• dials.search_beam_centre: Dramatically improved execution time for large data sets (#1612)
• dials.reindex: Write .refl file output in the default "MessagePack" format for better compatibility with downstream programs (#1616)
• dials.scale: Fix rare memory crash from infinite loop, that could occur with very bad quality datasets (#1622)

Improved Documentation

• dials.refine: More informative error message when reflections have weights of zero (#1584)

DIALS 3.3.3 (2021-02-15)

No changes to core DIALS in 3.3.3.

DIALS 3.3.2 (2021-02-01)

Bugfixes

• Remove unnecessary call to imageset.get_raw_data() while generating masks. This was causing performance issues when spotfinding. (#1449)
• dials.export: Allow data with either summation or profile fitted intensities to be exported. Previously, both were (erroneously) required to be present. (#1556)
• dials.scale: Fix crash if only summation intensities present and intensity_choice=combine (#1557)
• Fix unicode logging errors on Windows (#1565)

DIALS 3.3.1 (2021-01-18)

Features

• dials.index: More verbose debug logs when rejecting crystal models that are inconsistent with input symmetry (#1538)

Bugfixes

• dials.stills_process: Fix spotfinding error "Failed to remap experiment IDs" (#1180)
• Improved spotfinding performance for HDF5 when using a single processor. (#1539)

DIALS 3.3.0 (2021-01-04)

Features

• DIALS is now using GEMMI. (#1266)
• Upgrade h5py requirement to 3.1+ for SWMR-related functionality. (#1495)
• Added support for small integer types to DIALS flex arrays. (#1488)
• dials.estimate_resolution: Only use cc_half in default resolution analysis. (#1492)
• dials.export: Allow on-the-fly bzip2 or gzip compression for mmCIF output, because unmerged mmCIF reflection files are large. (#1480)
• dials.find_spots and dials.integrate both now have nproc=Auto by default, which uses the number of allowed/available cores detected. (#1441)
• dials.merge: Report <dF/s(dF)>, if anomalous=True. An html report is also generated to plot this statistic. (#1483)
• dials.scale: Apply a more realistic initial error model, or load the existing error model, if rescaling. (#1526)
• dials.stills_process: allow using different saturation cutoffs for indexing and integration. Useful for using saturated reflections for indexing while still rejecting them during integration. (#1473)

Bugfixes

• Internal: Logging metadata is now preserved when running spotfinding and integration across multiple processes. (#1484)
• Fix NXmx behaviour with h5py 3.1. (#1523)
• dials.cosym: Choose the cluster containing the most identity reindexing ops by default. Under some circumstances, particularly in the case of approximate pseudosymmetry, the previous behaviour could result in reindexing operators being chosen that weren't genuine indexing ambiguities, instead distorting the input unit cells. (#1514)
• dials.estimate_resolution: Handle very low multiplicity datasets without crashing, and better error handling. (#1494)
• dials.export,``dials.two_theta_refine``: Updates to mmcif output to conform to latest pdb dictionaries (v5). (#1528)
• dials.find_spots: fix crash when nproc=Auto. (#1019)
• dials.image_viewer: Fix crash on newer wxPython versions. (#1476)
• dials.index: Fix configuration error when there is more than one lattice search indexing method. (#1515)
• dials.merge: Fix incorrect output of SigF, N+, N- in merged.mtz. (#1522)
• dials.reciprocal_lattice_viewer: Fix error opening with wxPython 4.1+. (#1511)
• dials.scale: fix issues for some uses of multi-crystal rescaling if full_matrix=False. (#1479)

Improved Documentation

• Update information on how to care for an existing development environment, and remove outdated information. (#1472)
• Each of the available indexing strategies in dials.index now has some help text explaining how it works. You can view this help by calling dials.index -c -a1 -e1 and looking for method under indexing. (#1519)
• Include __init__ methods in autodoc generated library documentation. (#1520)
• dials.estimate_resolution: Improved documentation. (#1493)

Deprecations and Removals

• dials.algorithms.spot_finding.finder.SpotFinder: Use of __call__ to run spotfinding has been deprecated in favor of SpotFinder.find_spots(experiments). (#1484)

Misc

• #1469, #1481, #1484, #1487, #1491, #1496, #1497, #1498, #1499, #1500, #1501, #1514.

DIALS 3.2.3 (2020-12-07)

Bugfixes

• dials.slice_sequence: Fix crash using block_size= option (#1502)
• dials.scale: Fix broken exclude_images= option (#1509)

DIALS 3.2.2 (2020-11-23)

Bugfixes

• Fix case where dials.stills_process could swallow error messages
• dials.cosym: Fix non-determinism. Repeat runs will now give identical results. (#1490)
• Developers: Fix precommit installation failure on MacOS (#1489)

DIALS 3.2.1 (2020-11-09)

3.2 Branch releases will now use a fixed conda environment. This release is the first to use the same versions of all dependencies as 3.2.0.

Bugfixes

• dials.symmetry, dials.cosym and dials.two_theta_refine: Lowered default partiality_threshold from 0.99 to to 0.4. The previous default could occasionally result in too many reflections being rejected for particularly narrow wedges. (#1470)
• dials.stills_process Improve performance when using MPI by avoiding unnecessary log file writing (#1471)
• dials.scale: Fix scaling statistics output of r_anom data. (#1478)

DIALS 3.2.0 (2020-10-27)

Features

• DIALS development environments are now running Python 3.8 by default. (#1373)
• Add a scaled flag to the reflection table. Indicates which reflections are good after the scaling process. (#1377)
• Python warnings are now highlighted on the console log and written to log files (#1401)
• Exit error messages from commands will now be colourized (#1420)
• Change the way dials.integrate splits data into blocks, to reduce unnecessary data reads, increasing performance up to 35% in some cases (#1396)
• Add dials.util.mp.available_cores function (#1430)
• dials.refine: Trimming scans to observations for scan-varying refinement can now be turned off, using the parameter trim_scan_to_observations=False (#1374)
• dials.refine: Change default to separate_panels=False. This speeds up outlier rejection for multi-panel detectors. For metrology refinement this should be set to True (#1424)
• dials.merge: Add best_unit_cell option. If the best_unit_cell option is set in dials.scale, this will now propagate to the merged mtz output file. (#1444)
• DIALS bootstrap now allow creating a Python 3.9 environment (#1452)
• DIALS now uses pytype for limited static type checking. We hope that this will, over time, improve code quality. (#1364)
• dials.stills_process: Added process_percent= to restrict processing to a subset of data, sync reference geometry instead of overwriting it and handle composite spotfinding modes. (#1409)

Bugfixes

• dials.stills_process: Prevent memory usage getting too high by clearing the imageset cache during processing. (#1412)
• dials.find_spots_server: Return HTTP 500 instead of 200 when running fails (#1443)
• dials.find_spots_server: Fix multiprocessing-related crash on macOS with Python3.8 (#1447)
• dials.integrate: Fix failures when building with GCC 9 (#1456)
• dials.image_viewer: Fix deprecation warnings (#1462)
• dials.index: When using local index assignment, take into account phi in nearest neighbour analysis. This can significantly improve indexing rates in some cases with scans > 360° (#1459)
• dials.reindex: Show an error instead of crashing for bad reindex operations. (#1282)

Deprecations and Removals

• dials.refine: the parameter trim_scan_edges is renamed scan_margin and the former name is deprecated (#1374)
• The developer command dev.dials.show_test_failure_reasons was removed. (#1436)
• Remove clipper sources from new development installations (#1437)

Misc

• #1175, #1337, #1354, #1379, #1381, #1400, #1406, #1416, #1423, #1426, #1432, #1433, #1435, #1446, #1454, #1466, #1468

DIALS 3.1.4 (2020-10-12)

No changes to core DIALS in 3.1.4.

DIALS 3.1.3 (2020-09-28)

Bugfixes

• dials.integrate: fix integrator=3d_threaded crash if njobs > 1 (#1410)
• dials.integrate: Check for and show error message if shoebox data is missing (#1421)
• dials.refine: Avoid crash for experiments with zero reflections if the auto_reduction.action=remove option was active (#1417)

Improved Documentation

• dials.merge: improve help message by adding usage examples (#1413)
• dials.refine: More helpful error message when too few reflections (#1431)

DIALS 3.1.2 (2020-09-14)

Features

• dials.stills_process: Add parameter max_images= to limit the number of processed images

Bugfixes

• dials.integrate: fix crash when run with integrator=3d_threaded (#1404)
• dials.integrate: Minor performance improvements (#1399)
• dials.stills_process: MPI performance improvements for large datasets
• dials.stills_process: Fix error when using split logs

DIALS 3.1.1 (2020-09-01)

Bugfixes

• dials.scale: Prevent discarding of resolution limits in rare cases, which could cause incorrect symmetry determination, and worse results. (#1378)
• dials.cosym: filter out experiments with inconsistent unit cells (#1380)
• Internally slicing experiments now works if image range doesn't start at 1 (#1383)
• Restore missing I/sigma(I) resolution estimate log output (#1384)
• dials.image_viewer: "Save As" button now works, for single panels
• Fix developer libtbx.precommit installation error (#1375)

DIALS 3.1 (2020-08-17)

Features

• Supports Python 3.7 and 3.8. Python 3.6 remains the default. (#1236)
• Switch DIALS environment to use conda compilers. For development environments, a new dials script, located above the build directory, replaces the existing 'setpaths'-family of scripts. This means that all commands within the conda environment will now be available. (#1235)
• New command: dials.missing_reflections to identify connected regions of missing reflections in the asymmetric unit. (#1285)
• Improvements to image stacking in dials.image_viewer: - add pull-down selector for stacking mode - add modes for mean and max - add command-line selection for stacking mode - rename sum_images command-line option to stack_images (#1302)
• Reduce volume of output in dials.integrate; histograms and other less important information only shows in debug output. Pass the -vv option to restore the previous behaviour (#1319)
• dials.integrate: Experimental feature: Specifying output_unintegrated_reflections=False discards unintegrated data from output reflection file, for smaller output and faster post-processing (#1343)
• Rename dials.resolutionizer command to dials.estimate_resolution, and includes a html report. Writing png plot output is now turned off by default (passing plot=True will restore this behaviour). (#1330)
• dials.scale now separates anomalous pairs during error model analysis (#1332)
• dials.background: Add parameter corrected= to optionally use pedestal-and-gain corrected data (#1348)
• dials.combine_experiments: Add option output.max_reflections_per_experiment=, to reject experiments with too many reflections (#1369)

Bugfixes

• dials.image_viewer: Shoeboxes are now shown rotated with rotated detector panels. (#1189)
• dials.index: In multi-lattice indexing, ensure that reflections where refinement fails are flagged as unindexed. (#1350)
• dials.scale: Reflections excluded from scaling are no longer permanently excluded from any subsequent dials.scale jobs. (#1275)
• dials.scale: When using intensity_choice=combine (the default), don't exclude reflections that only have one of summed or profiled intensities available, but not both. (#1300)
• dials.split_experiments: Don't generate extra leading zeros in the output filename when not required e.g. output_09.expt -> output_9.expt (#1316)
• dials.plot_reflections: Fix invisible white spots on white background. (#1346)

Deprecations and Removals

• dials.find_spots: Deprecate spotfinder.filter.use_trusted_range= (#1156)
• setpaths.sh and related scripts in newly created DIALS development environments are made obsolete and will no longer work. (#1235)
• dials.show: Remove show_image_statistics= parameter. Use image_statistics.show_raw= for equivalent output (#1306)
• Log files will omit timings unless the relevant dials program was run with -v (#1313)

Misc

• #1184, #1216, #1288, #1312, #1320, #1322, #1325, #1328, #1352, #1365, #1366, #1370

DIALS 3.0.4 (2020-07-20)

• dials.scale: Allow usage of mode=image_group with filtering.method=deltacchalf when only providing a single data set (#1334)
• dials.import: When using a template and specifying an image_range, missing images outside of the range will not cause a failure (#1333)
• dials.stills_process: Show better error message in specific spotfinding failure case (#1180)

DIALS 3.0.3 (2020-07-06)

Features

• Developer tool: On posix systems, sending SIGUSR2 to DIALS commands will now print a stack trace (#1277)

Bugfixes

• HTML reports: Plot bin centres instead bin minimum for d_min line plots vs. resolution (#1323)
• dials.export: Fix inconsistency in mtz export when given a non-reference (e.g. I2 or primitive) setting (#1279)
• dials.refine_bravais_settings: Fix crash with large (>2gb) reflection tables and reduce memory use (#1274)
• dials.scale: Fix bug in outlier rejection code causing misidentification of outliers (with outlier_rejection=standard).
• dials.scale: Fix outlier rejection formula to avoid overconfidence in spuriously low values

DIALS 3.0.2 (2020-06-23)

Bugfixes

• Fix crash in scaling error model handling (#1243)

DIALS 3.0.1 (2020-06-11)

Features

• dials.reciprocal_lattice_viewer: Add an option to show lattice(s) in the crystal rather than laboratory frame. (#1259)
• Support for mtz project_name in export and scaling

Bugfixes

• dials.reciprocal_lattice_viewer: fix multiple experiment view for integrated data (#1284)

DIALS 3.0 (2020-05-22)

Features

• Show more useful output when crashing in C++ code (#659)
• dials.image_viewer: for the unit cell tool, rename parameters for consistency and add a new show_hkl option to filter displayed powder rings to select only those of interest. (#1192)
• In dials.integrate: changed the background box size multiplier to be a parameter (sigma_b_multiplier) - setting to small values significantly reduces memory requirements. (#1195)
• dials.image_viewer: add an overlaying showing pixels marked as strong by the spot-finding operations. That is, the pixels picked out by the "threshold" image. (#1200)
• dials.scale report file was renamed from scaling.html to dials.scale.html dials.symmetry report file was renamed from dials-symmetry.html to dials.symmetry.html (#1202)
• dials.report output file was renamed from dials-report.html to dials.report.html (#1206)
• dials.image_viewer: faster navigation between different image types. (#1213)
• Crystal model now has a new recalculated_unit_cell attribute. This allows it to store a post-refined unit cell (e.g. from dials.two_theta_refine) in addition to that from traditional geometry refinement (which was used for prediction). Downstream programs such as dials.scale and dials.export will now use the recalculated unit cell where appropriate. (#1214)
• New best_monoclinic_beta parameter for dials.refine_bravais_settings and dials.symmetry. Setting this to False will ensure that C2 is selected in preference to I2, where I2 would lead to a less oblique cell (i.e. smaller beta angle). (#1226)
• New scaling model, model=dose_decay, implementing a shared exponential decay component for multicrystal experiments (#1183)

Bugfixes

• Avoid empty "Unable to handle" messages on failed dials.import (#600)
• Functions from dials.export now raise exceptions on errors rather than exit. This improves their use elsewhere (such as in dials.scale). (#1205)
• Ensure dials.index chooses the C2 setting with the smallest beta angle (#1217)
• Fix propagation of best_unit_cell and application of resolution cutoffs in dials.scale and export_mtz. Add a new mtz.best_unit_cell parameter to dials.export (#1248)
• Make some of the DIALS tools furthest downstream (dials.scale, dials.symmetry, dials.merge and dials.report) more robust in the case of very few reflections. (#1263)

Misc

• #1221

DIALS 2.2 (2020-03-15)

Features

• dials.image_viewer: Add a choice between displaying the raw or the corrected image. (#634)
• Automatically choose between the simple and glm background determination algorithms, depending on whether the detector appears to be integrating or counting. (#706)
• Allow adjustment of font size for overlay text, such as Miller indices and resolution ring values. (#1074)
• Keep goniometer and scan objects in indexing of still data, if provided in input (#1076)
• Experimental: dials.image_viewer can be remotely controlled via a ZeroMQ endpoint with the zmq_endpoint PHIL parameter. Initially, the viewer can be commanded to load new images. This requires the (optional) ``pyzmq``package. (#1085)
• Programs now generate a unique identifier for each experiment created, and reflection tables are linked via the experiment_identifiers map (#1086)
• Introduce dials.anvil_correction to correct the absorption of the incident and diffracted X-ray beam by the diamond anvils in a pressure cell. Call dials.anvil_correction on the output of dials.integrate and then proceed to use post-integration tools as normal, just as though the sample had been measured in air. (#1090)
• Map of detector efficiency for photon counting detectors as a function of detector position added to report, based on the qe value applied at the end of integration. (#1108)
• Significantly reduce the amount of memory required to write .refl output files (#1115)
• Add maximum_trusted_value=N option to spot finding to temporarily allow override of trusted range, e.g. to find overloaded spots in spot finding. (#1157)
• array_family.flex interface has changed: background and centroid algorithms are set via public properties. Instead of flex.strategy use functools.partial with the same signature. as_miller_array() raises KeyError instead of Sorry. .extract_shoeboxes() lost its verbosity parameter, use log levels instead. (#1158)
• dials.stills_process now supports imagesets of length > 1 (e.g. grid scans) (#1174)

Bugfixes

• Fixed prediction on images numbered zero, so integrating works correctly. (#1128)
• Fix an issue (#1097) whereby aggregating small numbers of reflections into resolution bins could sometimes result in empty bins and consequent errors. (#1130)
• Ensure that restraints are ignored for parameterisations that are anyway fixed (#1142)
• Fix dials.search_beam_centre to ensure that the correct detector models are output when multiple detector models are present in the input. Fix dials.search_beam_centre n_macro_cycles option (previously it was starting from the original geometry every macro cycle). (#1145)
• dials.find_spots_server no longer slows down 3x when using resolution filters (#1170)

Misc

• #932, #1034, #1050, #1077

DIALS 2.1 (2019-12-12)

Features

• We now fully support Python 3 environments.
• MessagePack is now the default reflection table file format. Temporarily, the environment variable DIALS_USE_PICKLE can be used to revert to the previous pickle-based format, however this will be removed in a future version. (#986)
• new option for dials.show 'show_shared_models=True' displays which beam, crystal, and detector models are used across experiments (#996)
• Import still image sequence as N experiments dereferencing into one image set rather than one experiment. (#1014)
• Add reflection_table.get method for defaulted column access (#1031)

Bugfixes

• Don't use -2 to indicate masked pixels, except for DECTRIS detectors where this is to be expected. (#536)
• No longer show pixels that are above the trusted range upper bound as "saturated" on the "variance" image. (#846)
• Correctly account for scan-varying crystals while providing a scan range to dials.integrate (#962)
• Ensure that generated masks do not include pixels that are overloaded on a few images, but only pixels that are always outside the trusted range. (#978)
• Rewritten parameter auto-reduction code for dials.refine provides finer-grained fixing of individual parameters rather than whole parameterisations and correctly takes constrained parameters into account (#990)
• Fix output of predictions in dials.refine. A recently-introduced bug meant that the updated predictions weren't being copied to the output reflections file. (#991)
• Allow scan-varying refinement where either the crystal cell or orientation is fixed. (#999)
• Respect batch= option to dials.symmetry - can reduce time taken for finding the symmetry for large data sets. (#1000)
• Scan-varying refinement no longer fails when the scan is wider than the observed reflections (e.g. when the crystal has died). Instead, the scan is first trimmed to match the range of the diffraction. (#1025)
• If convert_sequences_to_stills then delete the goniometer and scan. (#1035)
• Correctly account for scan-varying crystals in dials.slice_sequence (#1040)
• Eliminate systematic absences before applying change of basis op to minimum cell in dials.symmetry. (#1064)

Improved Documentation

• Add "Extending DIALS" page to developer documentation (#893)

Deprecations and Removals

• The command dials.analyse_output was removed. Its replacement, dials.report, will give you more useful output. (#1009)

Misc

• #983, #1004

DIALS 2.0 (2019-10-23)

Features

• Support exporting multi-dataset and still experiments to XDS_ASCII (#637)
• Replace default spotfinder with improved dispersion algorithm (#758)
• dials.report now displays oscillation data with units and more significant figures (#896)
• A new program, dials.sequence_to_stills to create split a sequence into a separate still Experiment for every scan point in the sequence, splitting reflections as necessary. (#917)
• Moved dials.export format=best to dials.export_best as that one needed access to the format object, the rest do not, and having dials.export work in the general case seems like a better idea... (#921)
• Unified logging output for dials programs - logs are no longer split into .log and .debug.log. Use -v to get debug output. (#923)
• New command dials.resolutionizer (replaces xia2.resolutionizer). Add support for expt/refl in dials.resolutionizer. (#933)
• Changed the selection of reflections used for determination of the reflection profile parameters in integration. Now uses reflections which were previously used in refinement rather than all reflections, resulting in a speed improvement for large data sets and a negligible difference in the quality of the integrated results. (#942)
• dials.image_viewer now allows the choice between dispersion_extended (new default) and dispersion (old default) thresholding algorithms for investigating the effect of different spot-finding parameters. (#948)
• dials.rs_mapper now respects masked regions of images (including the trusted range mask). (#955)

Bugfixes

• Fix and reinstate normalisation option in dials.option (#919)

Misc

• #795, #862, #895, #915, #924