Switch dxtbx to unified dependency system #753
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is an attempt to consolidate the requirements for specifying dxtbx dependencies for development, testing, conda-forge, mac, linux, windows, prebuilt-cctbx and bootstrap. It is intended to encompass DIALS also, but dxtbx being handled is a prerequisite for updating DIALS, and using just for CI here seemed like a reasonable stepping point.
All dependencies are declared in a single
dependencies.yaml
in the dxtbx root. They are now split up into categories matching the conda-build definitions:build
is for build tooling only required when you build the packages (the user would not install these)host
is.. slightly harder to pin down. It's effectively the binary packages that you need present at build time, to e.g. link against.run
is any dependencies only required to actually run dxtbx, separate from building ittest
is any dependencies used for running tests (e.g. dials-data, pytest...)Platform/target selection of dependencies is handled with selectors. This is a very small subset of the concept used by conda-build. Effectively, you can mark lines as only included when a condition is met. For example:
pycbf
will only be included when using prebuilt (e.g. from conda-forge) cctbx.scipy
andpytest
will always be included,wxpython
will have different constraints depending on whether you are running on macOS or not, andpytest-nunit
will only be included on windows.Valid selector combinations are currently specified in the header of
dependencies.yaml
.The driver for this is continually having to update dependencies in several places whenever something needs to be changed. As well as separate dependency lists for mac, linux and windows, we have some custom heuristics for removing unnecessary packages from release bundles, we have the dependencies entirely separately specified as part of the conda-forge package, and it's otherwise pretty difficult to work out what is needed for build vs runtime (which lets us deliberately include less in the bundles).
Although this is a somewhat complex change, it isn't really adding complexity that isn't present across the system anyway.