Rework the documentation for poetry manager

[ddluke]

Tell us more.

Hi folks! First, I want to say that Renovate is a great tool, and it helped us a lot in our distributed repository stack
to automate dependency updates for a lot of use cases (terraform providers, docker image versions, GitHub action
versions and a lot more) 🚀 💪

However, we still have not adopted Renovate for any of our poetry managed python projects (albeit we would like to) due
to:

• lack of documentation
• partially confusing documentation
• missing configuration options
• irritating renovate behaviour

And I do see that there might be a general misconception of what Renovate should do with poetry managed python projects,
and what Renovate should or must not do. I want to say that I'm not a contributor of poetry myself, but I do have
roughly a decade python programming experience in diverse distributed cloud stacks and am a user of poetry since roughly
4-5 years.

There are a handful of notes in the Renovate docs that heavily irritate me, perhaps you could clarify these or rework the documentation?

In essence, what confuses me the most is actually all contained in this section of the documentation.

Renovate cannot accurately update locked versions of Poetry dependency ranges due to limitations in Poetry. For example, if the pyproject.toml has a constraint like coverage = "^7.2", and the version ion poetry.lock is 7.4.1, and we know that 7.4.3 is available, then Renovate can only run poetry update --lock --no-interaction coverage and hope the result is 7.4.3. Poetry does not support updating to a specific/exact version with the update command, and the above update command may not even update at all sometimes. For this reason it's much better to pin dependency versions in pyproject.toml, such as coverage = "7.4.1" because it then gives Renovate more control and the ability to accurate upgrade dependencies in increments like 7.4.1 to 7.4.3.

Complexity of dependency graphs

Renovate can only run poetry update --lock --no-interaction coverage and hope the result is 7.4.3

This is because dependency graphs are complex, and poetry respects the fact that this is the case. An update to 7.4.3
might as well just completely break the library or application, and if poetry refuses to update the dependency to 7.4.3
than this is probably for good reasons. One example could be that any of potentially hundreds of different and partially
interconnected dependencies in the dependency tree of the library may be marked as incompatible with 7.4.3. Naively or
forcefully updating the dependency to 7.4.3 in that case might completely break the application or library since it
becomes uninstallable.

Obviously, also poetry may contain bugs, but in general, if poetry refuses to choose version 7.4.3, one should accept that this is for good reasons.

Library vs Application Code

For this reason it's much better to pin dependency versions in pyproject.toml, such as coverage = "7.4.1" because it then gives Renovate more control

This is bad practice! Or at least it depends on whether the project contains library or application code. For
application code, this statement might make sense, since you really want to be sure that your code is deployed and
shipped with a deterministic dependency set. However, this is why lock files exist (poetry.lock)! Pinning dependencies
as general advice in the pyproject.toml file is bad even for application code. This is why you use lock files.

In fact, I would say that there are only a handful of cases where pinning dependencies for application code might make
sense:

• private dependencies, those definitely should be pinned down to patch level, be it only to reduce the risk of package
  namespace collision exploits when using multiple repository sources (e.g. PyPI + an additional private index)
• cloud provider constraints, e.g. when you know that your cloud provider only supports this exact patch version of some
  library (e.g. apache-airflow in AWS MWAA or GCP Cloud Composer, or apache-spark in AWS Glue)

For library code, this statement makes no sense at all. If every python library out there would follow this advice, then
no library would be installable anymore. The pyproject.toml is not a lock file, it contains package metadata. For
library code, you want to make sure that a downstream user of the library is give the freedom to pick a range of
versions of a given dependency, to avoid dependency clashes with other dependencies. If one would follow the advice of
Renovate for library code, and, to stick with the example, pin coverage to version 7.4.3, then every downstream library
would be forced to use this exact same version as well.

This is why, for library code, you usually don't want to pin dependencies to patch level. As a best practice, you
instead specify a range of versions that your library is compatible with, you add good coverage for these supported
versions, and allow a downstream consumer of your library to pick one of the supported dependency versions.

In fact, I wonder if Renovate should touch a pyproject.toml file at all, or if, then probably only for an explicit
configurable allow list (since it's obviously impossible for Renovate to differentiate between library and application
code).

Missing documentation on the picked poetry version

This is another thing that worries me. When I tested Renovate on some of our projects, I do constantly see changes like
these in lock files:

- # This file is automatically @generated by Poetry 1.6.1 and should not be changed by hand.
+ # This file is automatically @generated by Poetry 1.8.2 and should not be changed by hand.

This indicates that Renovate uses a version of poetry that differs from the version the lock file was generated with.
Could you please update the documentation to explain how the poetry version is chosen?

[rarkins]

This flow would be better:

1. Create a discussion per distinct topic
2. If you are given an answer, contribute a PR improving the docs

[jgen1]

Were there any discussions created as a result of this post @ddluke?