Break the DIALS/xfel circular dependency #627
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…bclass of cbf_wrapper
(IE don't crash dxtbx in the import step for FormatXTC)
This was referenced Apr 25, 2023
|
Explicitly requires cctbx/cctbx_project#872 to be merged |
|
Otherwise looks reasonable 👍 |
ndevenish
approved these changes
May 18, 2023
|
@ndevenish hey when cctbx/cctbx_project#872 and this PR are merged, the dxtbx and DIALS Azure pipelines will break until the next cctbx conda package release, end of September. Hence, I'll probably sit on this until about a week before the end of September to give things a chance to settle, meaning the DIALS/dxtbx pipelines will be brokeadoke for that week. Reasonable? How does that align with the DIALS release schedule? Thanks! |
|
I think we’ll have a DIALS patch release but that’s on a separate history now anyhow. Does cctbx have a nightly conda-build anywhere we could point at? |
|
Yep! https://github.com/cctbx/cctbx_project#nightly-builds
|
|
Okay, this looks like it's now pointing to the nightly CCTBX builds. Once cctbx/cctbx_project#872 is merged, this PR should pass tests (and thus be ready to merge). |
phyy-nx
added a commit
to cctbx/cctbx_project
that referenced
this pull request
Sep 26, 2023
Brief summary: * A new top level project in cctbx_project has been created: serialtbx * Any code needed by DIALS and dxtbx that was in xfel has been moved to serialtbx or directly into DIALS/dxtbx. That mostly includes code to read data at XFELs used by dxtbx, and stills-specific indexing/integration methods. * serialtbx has no imports of DIALS or dxtbx and so it can be directly configured by the conda packages cctbx and cctbx-base, @bkpoon. That said, there are no dispatchers or libtbx_config file so nothing specifically needs to be done to bootstrap.py, I believe. * All imports of xfel in DIALS and dxtbx have been updated to use the new serialtbx or DIALS/dxtbx imports * All other known repos that have xfel imports have been updated to use the new serialtbx or DIALS/dxtbx imports * xfel will remain configured in the DIALS binary bundles on dials.github.io and distributed by phenix. For conda, the plan is to make a cctbx.xfel conda package that includes all its dependencies (MPI, MySQL, etc.) Se also cctbx/dxtbx#627 and dials/dials#2404 Detailed listing of code movements: * xfel.util.jungfrau → serialtbx.detector From xfel.cftbx.detector: * cspad_detector → serialtbx.detector.legacy_metrology * generic_detector → serialtbx.detector.legacy_metrology * metrology → serialtbx.detector.legacy_metrology From xfel.mono_simulation: * max_like → serialtbx.mono_simulation * util → serialtbx.mono_simulation From xfel.cftbx.detector.cspad_cbf_tbx: * Constants moved to serialtbx.detector.cspad * get_psana_corrected_data → serialtbx.detector.cspad * cbf_wrapper → merged with dxtbx.format.FormatCBFMultiTile.cbf_wrapper * angle_and_axis → dxtbx.format.FormatCBFMultiTile.cbf_wrapper * center → serialtbx.detector * basis → serialtbx.detector * basis_from_geo → serialtbx.detector.xtc * read_slac_metrology → serialtbx.detector.cspad * add_frame_specific_cbf_tables → dxtbx.format.cbf_writer From xfel.command_line.cspad_detector_congruence: * iterate_detector_at_level → serialtbx.detector * iterate_panels → serialtbx.detector * id_from_name → serialtbx.detector * get_center → serialtbx.detector From xfel.command_line.frame_extractor: * ConstructFrame → serialtbx.util.construct_frame From xfel.cxi.cspad_ana.cspad_tbx: * Constants moved to serialtbx.detector.cspad * address_split → serialtbx.detector.xtc * dpack → serialtbx.detector.cspad * get_ebeam → serialtbx.detector.xtc * env_distance → serialtbx.detector.xtc * evt_wavelength → serialtbx.detector.xtc * env_detz → serialtbx.detector.xtc * old_address_to_new_address → serialtbx.detector.xtc From xfel.cxi.cspad_ana.rayonix_tbx.py: * Constants moved to serialtbx.detector.rayonix * get_rayonix_pixel_size → serialtbx.detector.rayonix * get_rayonix_detector_dimensions → serialtbx.detector.rayonix * get_data_from_psana_event → serialtbx.detector.rayonix C++ code from xfel/ext.cpp: * radial_average → dxtbx.ext From xfel.util: * sublattice_helper → dials.algorithms.integration Commits: * Move xfel.mono_simulation.max_like to serialtbx * Move time functionality from cspad_tbx to serialtbx/util/time.py * Move xfel.mono_simulation.util to serialtbx/mono_simulation * xfel.util.sublattice_helper moved to dials.algorithms.integration * Move xfel.command_line.frame_extractor.ConstructFrame to serialtbx.util * Move radial average c++ code to dxtbx * Move dpack to serialtbx.detector.cspad * Move jiffy functions from cspad_detector_congruence to serialtbx.detector * Move utility functions and constants out of cspad_tbx into serialtbx.detector.cspad and serialtbx.detector.xtc * Move movement of functions into serialtbx * Moving out of cspad_cbf_tbx: - Into serialtbx.detector: basis, center - Into serialtbx.detector.cspad: read_slac_metrology - Into serialtbx.detector.xtc: basis_from_geo * Use dxtbx's cbf_wrapper by adding in the only function in the xfel subclass of cbf_wrapper * Move add_frame_specific_cbf_tables to dxtbx * Move xfel.util.jungfrau to serialtbx.detector * cspad constants and data reading moved to serialtbx * Move rayonix functions and constants into serialtbx * Move old_address_to_new_address to serialtbx * Move legacy image pickle metrology code to serialtbx.detector.legacy_metrology * Missing imports and a few more functions moving into serialtbx * Temporarily check out serial_tbx branches on other repos * Duplicated code * Fix import * Add serialtbx to CCIBuilder configuration list. This should ensure serialtbx is included in the conda cctbx-base package * Revert "Temporarily check out serial_tbx branches on other repos" --------- Co-authored-by: Billy K. Poon <bkpoon@lbl.gov>
phyy-nx
added a commit
to dials/dials
that referenced
this pull request
Sep 26, 2023
DIALS and dxtbx have had dependencies on cctbx_project/xfel, and vice versa. This PR is one of 8 that will break the cyclic dependency. Notably for DIALS, this specifically moves sublattice_helper from xfel to dials.algorithms.integration and removes: @pytest.mark.xfel For more detail, see cctbx/cctbx_project#872 and cctbx/dxtbx#627 Closes #2076 and #2389 Commits: * max_like moved from xfel to serialtbx * Time code moved from xfel to serialtbx * green_curve_area and ewald_proximal_volume moved to serialtbx * sublattice_helper moved in from xfel. Within dials.stills_process, create negative control for background modeling or validation. * Change import of sublattice_helper * ConstructFrame moved to serialtbx * Remove check for xfel in tests * Fix import * Remove pytest xfel mark * Temporarily checkout dxtbx and cctbx serial_tbx branches * Revert "Remove cctbx.xfel dependency from dials.index (#2390)" * Revert "Temporarily checkout dxtbx and cctbx serial_tbx branches" * Use latest CCTBX nightly for prebuilt --------- Co-authored-by: Nicholas Sauter <nksauter@lbl.gov> Co-authored-by: Nicholas Devenish <ndevenish@gmail.com>
toastisme
pushed a commit
to toastisme/dxtbx
that referenced
this pull request
Jul 18, 2024
DIALS and dxtbx have had dependencies on cctbx_project/xfel, and vice versa. This PR is one of 8 that will break the cyclic dependency.
Notably for dxtbx, this specifically removes:
pytest.importorskip("xfel")
For more detail, see cctbx/cctbx_project#872 and dials/dials#2404
Commits:
* Time code moved from xfel to serialtbx
* Move in radial_average c++ code from xfel
* dpack moved to serialtbx
* Jiffy function iterate_detector_at_level moved from xfel to serialtbx
* Utility functions and constants from cspad_tbx moved to serialtbx
* basis_from_geo and basis moved into serialtbx
* Use dxtbx's cbf_wrapper by adding in the only function in the xfel subclass of cbf_wrapper
* Move in add_frame_specific_cbf_tables
* xfel.util.jungfrau moved to serialtbx.detector
* cspad constants and data reading moved to serialtbx
* Rayonix toolbox methods and constants moved to serialtbx
* Legacy image pickle metrology code moved to serialtbx.detector.legacy_metrology
* Remove xfel import tests
* Better handle if SIT_* variables are missing for psana (IE don't crash dxtbx in the import step for FormatXTC)
* Fix leftover conflict
* Remove xfel reference that snuck in with nxmx_writer
* Use CCTBX nightly until serialtbx hits stable cctbx
* Remove importer skip for xfel from test_nxmx_writer
---------
Co-authored-by: Nicholas Devenish <ndevenish@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
DIALS and dxtbx have had dependencies on cctbx_project/xfel, and vice versa. This PR is one of 8 that will break the cyclic dependency.
Notably for dxtbx, this specifically removes:
pytest.importorskip("xfel")For more detail, see cctbx/cctbx_project#872 and dials/dials#2404 (placeholder)
Closes #239