Replace `poetry` with `pip`, build with `hatchling`, dynamically calculate minimum dependencies with `uv` #733

mfisher87 · 2024-06-30T04:01:41Z

"Dependency groups" are a poetry-specific feature that's different from PEP621 "optional dependencies", aka "extras". To make things compatible with other ways of dependency management, switch to PEP621 optional dependencies, but specified using the Poetry configuration aliases to retain support for poetry. While the Poetry "dependency groups" functionality may be more aimed at development workflows, it's not compatible with other tools and in my experience the dominant convention in the ecosystem is to use optional dependencies. As @itcarroll points out below, Hatch supports a similar concept called "environment dependencies" and specifies them with a different tool-specific configuration key. PDM also calls it something different, "dev dependencies", and uses another unique config table. 😓

I really don't like the way poetry requires optional dependencies to be specified. It requires duplication.

Thoughts? I don't want to add all this duplication to our codebase. Can we go forward with replacing Poetry with good ol' setuptools or maybe hatchling for builds and switch the optional dependency groups to the standard format? Anything PEP621 compatible would be a step forward IMO. I can take that on as part of this PR. The PEP621 example shows our use case, test dependencies, implemented in the conventional way.

Closes #734
Closes #619
Closes #374

📚 Documentation preview 📚: https://earthaccess--733.org.readthedocs.build/en/733/

TODO:

Update docs

mfisher87 · 2024-06-30T05:04:36Z

.pre-commit-config.yaml

+ - repo: https://github.com/python-poetry/poetry
+ rev: '1.8.3'
+ hooks:
+ - id: poetry-check


this will fail if the lockfile hasn't been updated, or if there's a typo in a package name in poetry config.

itcarroll · 2024-06-30T17:58:27Z

While the Poetry "dependency groups" functionality may be more aimed at development workflows, it's not compatible with other tools and in my experience the dominant convention in the ecosystem is to use optional dependencies.

I have been exploring hatch w/ hatchling, and it is both PEP compliant and also (like Poetry) avoids using extras (a.k.a optional-dependencies) for managing development environments. Information about extras is built into the distribution metadata and primarily provides extra packages users might want (i.e. xarray[io] if you want netCDF4, etc.). Hatch (and Poetry) allow separate ways of providing packages for and setting up development environments; information not included in distribution metadata. I haven't been compelled to duplicate package lists for simple cases with Hatch. Note that the current potential downside with Hatch is a lack of lock files.

mfisher87 · 2024-06-30T18:08:09Z

Just to clarify, I'm for including these optional dependency groups in the package metadata, as that's in my experience a very common method and is compatible with all environment / package managers, and as far as I know there is no standard way to specify dependency groups that do not go in the package metadata.

I'd like to avoid pros and cons of environment / package managers here (i.e. Hatch, Poetry), as we can isolate that concern from our build tooling and configuration. Regardless of whether we decide down the road to document / encourage Hatch or Poetry or something else for package management, if our build configuration is PEP621 compliant, contributors can choose any tool.

mfisher87 · 2024-06-30T18:24:38Z

To make things more confusing, PDM has "dev dependencies", their own uniquely named version of Poetry's "dependency groups" and Hatch's "environment dependencies". Wow! The "same" thing three times with three names and three config keys.

I'd really like to treat this Python environment management feature as non-standard and avoid it until there's a unified interface, which I'm sure there's already an open PEP for 😆

mfisher87 · 2024-06-30T18:30:38Z

Found it: https://peps.python.org/pep-0735/

chuckwondo

@mfisher87, sorry this has remained unreviewed for so long! LGTM!

I'm approving, but do you need to update any "contributing" instructions regarding commands that a contributor needs to run to get their environment setup with dev/test deps?

mfisher87 · 2024-09-03T14:46:06Z

Thanks, Chuck! Good point, I do imagine I missed some stuff there.

TODO:

Ensure docs are up-to-date with dependency management changes

Poetry is required to use dependency groups. To make things compatible with other ways of dependency management, switch to PEP621 optional dependencies, but specified using the Poetry configuration aliases to retain support for poetry. TODO: Docs

As long as we're using it we should be validating! :)

Maybe we could use Nox instead of scripts at some point? Then the behaviors can be coupled to the correct environments.

These settings are already specified in pyproject.toml

Closes #734

mfisher87 · 2024-09-04T02:27:38Z

@chuckwondo @danielfromearth I committed the changes we were working on in the call (24a5191), and clearly they are no longer poetry compatible :) I didn't have time to debug yet!

danielfromearth · 2024-09-04T17:32:16Z

So, where do we want to go from here? If there's no way to remove the redundant dependency specifications, then get this working with another manager, like hatch, and then drop the poetry usage?

pyproject.toml

danielfromearth · 2024-09-06T13:40:39Z

hmm, hadn't seen uv yet, but that looks quite promising!

It does look like the particular poetry PR for the project section dependencies (python-poetry/poetry#9135) might be close to being finalized (all tests are passing just marked ready-for-review 4 days ago).

mfisher87 · 2024-09-06T15:09:35Z

It does look like the particular poetry PR for the project section dependencies (python-poetry/poetry#9135) might be close to being finalized (all tests are passing just marked ready-for-review 4 days ago).

While it would be nice to not have to do the work to rip out poetry given this change ☝️ I still think we should rip out poetry. I see poetry as purely a source of complexity. E.g. why do we need ^ in Python? We have a compatible release specifier (~=), but we want it to feel more like Javascript? 🤷

Anthony Sottile (pre-commit): why I will never use python-poetry

This probably never worked? Kerchunk doesn't use calver.

- Replace Poetry - For dev, use `pip` & `venv` (users can of course use anything else that's standards-compliant, like `uv`) - For calculating minimum dependencies and reproducible locking, use `uv` - For build, use hatchling - Document usage of `pip` and `venv` in developer documentation. - Remove Makefile -- it's undocumented anyway! - Add Nox for task-running - Remove redundant scripts - Linting is run with pre-commit - Type checking is run with Nox - Update CI - Publish with PyPA official action

mfisher87 · 2024-09-07T17:54:43Z

.github/workflows/integration-test.yml

 jobs:
 integration-tests:
 runs-on: ubuntu-latest
 strategy:
 matrix:
- python-version: ["3.9", "3.10", "3.11", "3.12"]
+ python-version: ["3.9", "3.12"]


I felt it was a bit wasteful to run integration tests for every supported python version.

mfisher87 · 2024-09-07T17:56:00Z

.github/workflows/test-mindeps.yml

- - name: Install
- run: python -m pip install --no-deps -e .
+ - name: Install minimum-compatible dependencies
+ run: uv sync --resolution lowest-direct


lowest-direct matches what we were already doing with environment-mindeps.yml. lowest doesn't resolve successfully.

There are things other than uv that can do "lowest" resolution and lockfiles, I just reached for something I knew was fast and had single-file multi-platform lockfiles and wasn't poetry.

mfisher87 · 2024-09-07T17:57:13Z

Makefile

This file wasn't documented or used anywhere.

mfisher87 · 2024-09-07T17:59:03Z

docs/contributing/development.md

+If you don't have pipx (pip for applications), then you can install with
+`pip install pipx` (the only case were installing an application with regular
+pip is reasonable). If you use macOS, then pipx and nox are both in brew, use
+`brew install pipx nox`.


From Scientific Python project template's documentation

mfisher87 · 2024-09-07T18:01:09Z

docs/contributing/development.md

-`pyproject.toml`.
+If you need to add a new dependency, edit `pyproject.toml` and insert the dependency in
+the correct location (either in the `dependencies` array or
+`[project.optional-dependencies]`.


uv and pixi provide good add user experiences, without the need to do the weird replacement of ^s. If we want that UX, let's chat about which tool we want to endorse.

mfisher87 · 2024-09-07T20:09:04Z

ci/environment-dev.yml

To me, it doesn't make sense for this to be in the ci directory.

mfisher87 · 2024-09-07T23:43:02Z

.pre-commit-config.yaml

+ - repo: https://github.com/astral-sh/uv-pre-commit
+ rev: "0.4.7"
+ hooks:
+ - id: uv-lock


Lockfile stays up-to-date without user needing to learn to use any new tooling

mfisher87 force-pushed the poetry-groups-to-pep621-optional-deps branch 3 times, most recently from 439cb98 to faa4571 Compare June 30, 2024 04:50

mfisher87 commented Jun 30, 2024

View reviewed changes

mfisher87 marked this pull request as ready for review June 30, 2024 14:09

mfisher87 requested review from chuckwondo, betolink and jhkennedy June 30, 2024 14:09

mfisher87 mentioned this pull request Jul 6, 2024

Adopt nox #750

Draft

mfisher87 added the hackathon An issue we'd like to visit during a hackathon event label Jul 8, 2024

chuckwondo previously approved these changes Sep 3, 2024

View reviewed changes

mfisher87 added 9 commits September 3, 2024 12:34

Add new poetry hooks

f2e15ce

As long as we're using it we should be validating! :)

Update Python compatibility in README

732abd1

Attempt to fix tests by installing test dependencies

a31b7ae

Decouple test, lint, typecheck scripts

5704abb

Maybe we could use Nox instead of scripts at some point? Then the behaviors can be coupled to the correct environments.

Remove redundant config

82189e7

These settings are already specified in pyproject.toml

Avoid duplicate test workflow runs

5b819a3

Closes #734

Install docs dependencies during readthedocs build

21be5f2

Migrate poetry dependency specification to standardized specification

24a5191

mfisher87 dismissed chuckwondo’s stale review via 24a5191 September 4, 2024 02:25

mfisher87 force-pushed the poetry-groups-to-pep621-optional-deps branch from 9f18f69 to 24a5191 Compare September 4, 2024 02:25

itcarroll reviewed Sep 4, 2024

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

mfisher87 added 11 commits September 7, 2024 10:02

Replace caret dependency notation

edd6442

Fixup conditional dependency specifications

6ac66b7

Unpin kerchunk extra dependencies

a39a519

This probably never worked? Kerchunk doesn't use calver.

Calculate minimum dependencies dynamically with uv

cde7b85

Encode mypy's files and python version in config

5f5986f

Replace poetry in GitHub Actions

6dada1a

Replace poetry in conda dev environment

8e8a7e7

Replace poetry for binder

8643011

Replace poetry for readthedocs

bd3fe74

Add nox dev dependency and fixup noxfile

cadcbdd

mfisher87 commented Sep 7, 2024

View reviewed changes

Fixup composable action

e194f14

mfisher87 force-pushed the poetry-groups-to-pep621-optional-deps branch 3 times, most recently from 0e0549f to 20073a2 Compare September 7, 2024 18:14

Install test deps for mindeps test action

c0cf028

mfisher87 force-pushed the poetry-groups-to-pep621-optional-deps branch from 20073a2 to c0cf028 Compare September 7, 2024 18:33

Upgrade minimum evrsions of some test dependencies

9818907

mfisher87 force-pushed the poetry-groups-to-pep621-optional-deps branch from 6fd31a8 to 9818907 Compare September 7, 2024 18:41

mfisher87 changed the title ~~Specify dependency groups in a way that doesn't require poetry~~ Replace poetry with pip, add uv for dynamically calculating minimum dependencies Sep 7, 2024

mfisher87 changed the title ~~Replace poetry with pip, add uv for dynamically calculating minimum dependencies~~ Replace poetry with pip, build with hatchling, add uv for dynamically calculating minimum dependencies Sep 7, 2024

mfisher87 changed the title ~~Replace poetry with pip, build with hatchling, add uv for dynamically calculating minimum dependencies~~ Replace poetry with pip, build with hatchling, dynamically calculate minimum dependencies with uv Sep 7, 2024

mfisher87 commented Sep 7, 2024

View reviewed changes

ci/environment-dev.yml Outdated

Copy link

Member Author

mfisher87 Sep 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To me, it doesn't make sense for this to be in the ci directory.

mfisher87 requested review from itcarroll and chuckwondo September 7, 2024 23:40

Add uv lockfile pre-commit check

7c4eb2e

mfisher87 commented Sep 7, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace `poetry` with `pip`, build with `hatchling`, dynamically calculate minimum dependencies with `uv` #733

Replace `poetry` with `pip`, build with `hatchling`, dynamically calculate minimum dependencies with `uv` #733

mfisher87 commented Jun 30, 2024 •

edited

Loading

mfisher87 Jun 30, 2024

itcarroll commented Jun 30, 2024

mfisher87 commented Jun 30, 2024 •

edited

Loading

mfisher87 commented Jun 30, 2024 •

edited

Loading

mfisher87 commented Jun 30, 2024

chuckwondo left a comment

mfisher87 commented Sep 3, 2024 •

edited

Loading

mfisher87 commented Sep 4, 2024 •

edited

Loading

danielfromearth commented Sep 4, 2024

danielfromearth commented Sep 6, 2024

mfisher87 commented Sep 6, 2024

mfisher87 Sep 7, 2024

mfisher87 Sep 7, 2024

mfisher87 Sep 7, 2024

mfisher87 Sep 7, 2024

mfisher87 Sep 7, 2024

mfisher87 Sep 7, 2024 •

edited

Loading

mfisher87 Sep 7, 2024

mfisher87 Sep 7, 2024

Replace poetry with pip, build with hatchling, dynamically calculate minimum dependencies with uv #733

Are you sure you want to change the base?

Replace poetry with pip, build with hatchling, dynamically calculate minimum dependencies with uv #733

Conversation

mfisher87 commented Jun 30, 2024 • edited Loading

Choose a reason for hiding this comment

itcarroll commented Jun 30, 2024

mfisher87 commented Jun 30, 2024 • edited Loading

mfisher87 commented Jun 30, 2024 • edited Loading

mfisher87 commented Jun 30, 2024

chuckwondo left a comment

Choose a reason for hiding this comment

mfisher87 commented Sep 3, 2024 • edited Loading

mfisher87 commented Sep 4, 2024 • edited Loading

danielfromearth commented Sep 4, 2024

danielfromearth commented Sep 6, 2024

mfisher87 commented Sep 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mfisher87 Sep 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Replace `poetry` with `pip`, build with `hatchling`, dynamically calculate minimum dependencies with `uv` #733

Replace `poetry` with `pip`, build with `hatchling`, dynamically calculate minimum dependencies with `uv` #733

mfisher87 commented Jun 30, 2024 •

edited

Loading

mfisher87 commented Jun 30, 2024 •

edited

Loading

mfisher87 commented Jun 30, 2024 •

edited

Loading

mfisher87 commented Sep 3, 2024 •

edited

Loading

mfisher87 commented Sep 4, 2024 •

edited

Loading

mfisher87 Sep 7, 2024 •

edited

Loading