Breaking behavior for installation of root package if extension modules are present

[classner]

• Poetry version: 1.6.1 vs. 1.4.0

• Python version: 3.8.10

• OS version and name: Ubuntu 20.04

• pyproject.toml: NA

• I am on the latest stable Poetry version, installed using a recommended method.

• I have searched the issues of this repo and believe that this is not a duplicate.

• I have consulted the FAQ and blog for any relevant entries or release notes.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

Up until poetry 1.4.0, the root package was usually linked into the poetry virtual environment (e.g., using a .egg-link file). In poetry 1.6.1 I see that poetry installs (copies) the project and binary files into the virtual environment instead of installing it when running poetry install - however, when doing poetry run ... the content of the source folder takes precedence over the installed package.

This leads to many potential problems, but in particular binary build artifacts are missing at execution time (poetry run ...) since they are installed to the installation location in the virtualenv path instead of the local project folder.

I do not know exactly in which version between 1.4.0 (working build) to 1.6.1 (broken build) this change was introduced. A principled solution would be good here, the current situation can lead to a host of inconsistencies.

[dimbleby]

I can't tell what this is reporting.

What commands did you run? What happened? What did you expect to happen?

A docker file containing that sequence of commands and reproducing whatever the problem is would be a particularly clear way of sharing.

[dimbleby]

insofar as I can parse this, I guess it duplicates #8377

[classner]

Hi @dimbleby !
Thanks for your quick reply. It is definitely related to #8377, but goes a bit beyond that, I think. You can find a minimal reproducing package attached here: repro_poetry_1_6_0_build.zip. To reproduce: use poetry < 1.6 (tested with 1.4.0) and run poetry install, poetry run repro_run in the repo. This works and prints the location of the extension module. Use with poetry latest from PyPi and it fails to use the extension module.

On poetry version prior to 1.6, poetry install would install extension modules into the project tree directly. To properly work with these sources, it adds a .pth file into the virtual environment that points to that source tree. When doing a poetry run ... for any script entry point, everything works simply by setting up the right venv and then calling the script entry point. This automatically locates the sources through the .pth file; .py and extension modules are in the right spot and everything runs.

What I observe for poetry with version 1.6 is the following behavior: poetry install installs extension modules directly into the virtual environment. In fact, the entire project is installed into the virtual environment (but only if extension modules are present). I assume this is in fact erroneous behavior, because poetry still at the same time creates the .pth links to the original source tree as well. The original source tree is lacking the compiled extension modules now, though. If poetry run ... for any script entry point is executed, I assume it again activates the virtual environment. But thanks to the .pth file present there, it links back to the original source tree for execution and fails to run the script due to the missing compiled extension modules there.

In any case, the behavior in 1.6 is inconsistent with extension modules and that probably warrants another look. This also is a breaking change on a minor version update, which probably should be avoided. Is it a duplicate for #8377? Maybe, but 8377 is particularly concerned with pip install -e behavior - I'm not sure how that relates to the regular poetry install, poetry run workflow. Happy to continue the discussion there! Does the current proposed fix for 8377 also address these concerns?

[dimbleby]

The "proposed" fix is already merged: try it!

[classner]

The issue still persists as of today on 1.7.0.dev0 (7575763 on master) with the reproducer provided above.

[dimbleby]

The fix you should have tried is in poetry-core, the poetry revision is not relevant.

[dimbleby]

Hmm:

$ poetry run repro_run
Failure! Can not import extension module.
$ HERE=$PWD ; (cd .. && poetry --directory $HERE run repro_run)
Success! Found the module at /home/dch/.virtualenvs/foo/lib/python3.10/site-packages/repro_poetry_1_6_0_build/ext.cpython-310-x86_64-linux-gnu.so

it sems that this fails exactly when the current working directory is the project root.

You can see similar by activating the virtual environment directly and running python -c "import repro_poetry_1_6_0_build.ext"

if you can get to the bottom of why this works from everywhere except the project root, that'll be progress...

[classner]

Even in that case the behavior is presumably unintended: you can see that it loaded the copy of the package from within the virtual environment. That is 1) inconsistent with poetry’s behavior in general, if there are no extension modules there is no such copy and 2) this does not correspond to poetry’s intention to use the project folder, for which a .pth link file is created (which is ignored since the package is installed in-place).

I don’t have time to look into this further at the moment, but the reproducer should be a great starting point to figure out what’s happening here!

[dimbleby]

yeah, this is all #8377 all along then

you can verify by changing the build requirements in pyproject.toml:

requires = ["poetry-core@git+https://github.com/python-poetry/poetry-core.git", "setuptools"]

please close

[classner]

Perfect, closing!

[thebalaa]

As discussed in matrix-org/synapse#16147 this issue remains for packages that have compiled modules.

The issue appears to be that prior to 1.5.1 a compiled extension module would be copied into the local module directory however that is no longer the case in the latest versions of poetry, as far as I can tell python-poetry/poetry-core#633 did not resolve this issue

[dimbleby]

this issue is closed (and also a duplicate)

if you have a new problem, raise a new issue - with a repro

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.