Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch dxtbx to unified dependency system #753

Merged
merged 10 commits into from
Aug 23, 2024

Conversation

ndevenish
Copy link
Collaborator

This is an attempt to consolidate the requirements for specifying dxtbx dependencies for development, testing, conda-forge, mac, linux, windows, prebuilt-cctbx and bootstrap. It is intended to encompass DIALS also, but dxtbx being handled is a prerequisite for updating DIALS, and using just for CI here seemed like a reasonable stepping point.

All dependencies are declared in a single dependencies.yaml in the dxtbx root. They are now split up into categories matching the conda-build definitions:

  • build is for build tooling only required when you build the packages (the user would not install these)
  • host is.. slightly harder to pin down. It's effectively the binary packages that you need present at build time, to e.g. link against.
  • run is any dependencies only required to actually run dxtbx, separate from building it
  • test is any dependencies used for running tests (e.g. dials-data, pytest...)

Platform/target selection of dependencies is handled with selectors. This is a very small subset of the concept used by conda-build. Effectively, you can mark lines as only included when a condition is met. For example:

  - pycbf #[prebuilt_cctbx]
  - scipy
  - wxpython >=4.2.0 #[not osx]
  - wxpython >=4.2.0=*_5 #[osx]
  - pytest
  - pytest-nunit # [win]
  • pycbf will only be included when using prebuilt (e.g. from conda-forge) cctbx. scipy and pytest will always be included, wxpython will have different constraints depending on whether you are running on macOS or not, and pytest-nunit will only be included on windows.

Valid selector combinations are currently specified in the header of dependencies.yaml.

The driver for this is continually having to update dependencies in several places whenever something needs to be changed. As well as separate dependency lists for mac, linux and windows, we have some custom heuristics for removing unnecessary packages from release bundles, we have the dependencies entirely separately specified as part of the conda-forge package, and it's otherwise pretty difficult to work out what is needed for build vs runtime (which lets us deliberately include less in the bundles).


Although this is a somewhat complex change, it isn't really adding complexity that isn't present across the system anyway.

Copy link

codecov bot commented Aug 20, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 41.54%. Comparing base (38c95db) to head (64d11d9).
Report is 4 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #753   +/-   ##
=======================================
  Coverage   41.54%   41.54%           
=======================================
  Files         189      189           
  Lines       17060    17060           
  Branches     3271     3271           
=======================================
  Hits         7088     7088           
  Misses       9318     9318           
  Partials      654      654           

@ndevenish ndevenish marked this pull request as ready for review August 20, 2024 14:36
@ndevenish ndevenish merged commit 853bbb7 into cctbx:main Aug 23, 2024
14 checks passed
@ndevenish ndevenish deleted the unify_dependencies branch August 23, 2024 11:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant