In Github action, pip installs from pypi instead of dist

[ChristopherRabotin]

Hi there,

This is an odd behavior I haven't seen before. With the CI file generated by maturin, one of the steps includes pip install pkg_name --find-links dist --force-reinstall. This typically works without issue.

However, without changing the github action, a PR on my repo does not work anymore: pip now doesn't seem to find the package in the dist folder and instead downloads it from pypi it seems. This behavior happens on windows, mac, and linux.

The culprit is shown at the top of the build:

Looking in links: dist
Collecting nyx_space
  Downloading nyx_space-2.0.0a2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.2/34.2 MB 5.3 MB/s eta 0:00:00

Version 2.0.0a2 is indeed what's currently on pypi, but I am expecting the CI to install the package built for this run.

Any idea what could be happening?

Thanks

[ChristopherRabotin]

Hey,

Any idea on how I can debug this? I've tried reverting to a prior version of maturin, but that didn't do anything. I also tried using find to find the exact wheel, but that only works for the target where the first package found matches the current target.

It's a total blocker on any of my work, I wasted the whole weekend trying to fix this.

Seriously, at this stage, any tip would help.

Thanks

[messense]

Sorry for the late response, it's public holidays recently.

I'm not sure what changed, maybe Python version or pip version? Anyway you can try something like pip install --force-reinstall ./dist/nyx_space-*-cp310-cp310-*.whl instead.

[messense]

And if you want to debug it, try pip install pkg_name --find-links dist --force-reinstall -vv to enable verbose logging.

[ChristopherRabotin]

Thanks for getting back to me.
For some reason, pip seems to ignore the dist folder entirely:

set -e
  ls -lh dist
  pip install nyx_space --find-links dist --force-reinstall -vv
  pip install pytest numpy pandas plotly pyarrow scipy
  pytest
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.10.12/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.10.12/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.10.12/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.10.12/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.10.12/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.10.12/x64/lib
    CARGO_INCREMENTAL: 0
    CARGO_TERM_COLOR: always
    ACTIONS_CACHE_URL: https://artifactcache.actions.githubusercontent.com/PgOGOZmmfzSIDYThjpbBoGQYiv6G84AefcZNtc8OYSZvugDUnJ/
    ACTIONS_RUNTIME_TOKEN: ***
    SCCACHE_GHA_ENABLED: true
    RUSTC_WRAPPER: sccache
total 327M
-rw-r--r-- 1 runner docker 33M Jun 26 13:32 nyx_space-2.0.0a2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-r--r-- 1 runner docker 33M Jun 26 13:33 nyx_space-2.0.0a2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-r--r-- 1 runner docker 33M Jun 26 13:35 nyx_space-2.0.0a2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-r--r-- 1 runner docker 33M Jun 26 13:29 nyx_space-2.0.0a2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-r--r-- 1 runner docker 33M Jun 26 13:30 nyx_space-2.0.0a2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-r--r-- 1 runner docker 33M Jun 26 13:31 nyx_space-2.0.0a2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-r--r-- 1 runner docker 33M Jun 26 13:39 nyx_space-2.0.0a2-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-r--r-- 1 runner docker 33M Jun 26 13:36 nyx_space-2.0.0a2-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-r--r-- 1 runner docker 33M Jun 26 13:37 nyx_space-2.0.0a2-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-r--r-- 1 runner docker 33M Jun 26 13:38 nyx_space-2.0.0a2-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Using pip 23.1.2 from /opt/hostedtoolcache/Python/3.10.12/x64/lib/python3.10/site-packages/pip (python 3.10)
Non-user install because site-packages writeable
Created temporary directory: /tmp/pip-build-tracker-5occzdml
Initialized build tracking at /tmp/pip-build-tracker-5occzdml
Created build tracker: /tmp/pip-build-tracker-5occzdml
Entered build tracker: /tmp/pip-build-tracker-5occzdml
Created temporary directory: /tmp/pip-install-8fc1sc13
Created temporary directory: /tmp/pip-ephem-wheel-cache-das679tc
Looking in links: dist
1 location(s) to search for versions of nyx-space:
* https://pypi.org/simple/nyx-space/
Fetching project page and analyzing links: https://pypi.org/simple/nyx-space/
Getting page https://pypi.org/simple/nyx-space/
Found index url https://pypi.org/simple/
Looking up "https://pypi.org/simple/nyx-space/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pypi.org:443
[https://pypi.org:443](https://pypi.org/) "GET /simple/nyx-space/ HTTP/1.1" 200 12281
Updating cache with response from "https://pypi.org/simple/nyx-space/"
etag object cached for 1209600 seconds
Caching due to etag
(...)

Log: https://github.com/nyx-space/nyx/actions/runs/5378666633/jobs/9758777518?pr=191

[messense]

Found link file:///home/runner/work/nyx/nyx/dist/nyx_space-2.0.0a2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl, version: 2.0.0a2
Local files found: /home/runner/work/nyx/nyx/dist/nyx_space-2.0.0a2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Given no hashes to check 7 links for project 'nyx-space': discarding no candidates

Looks like pip found it but decided to discard it for some reason, maybe you should ask pypa/pip.

[ChristopherRabotin]

Oh, good find! Thanks, I'll ask them.

…

On Mon, Jun 26, 2023, 08:04 messense ***@***.***> wrote: Found link file:///home/runner/work/nyx/nyx/dist/nyx_space-2.0.0a2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl, version: 2.0.0a2 Local files found: /home/runner/work/nyx/nyx/dist/nyx_space-2.0.0a2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl Given no hashes to check 7 links for project 'nyx-space': discarding no candidates Looks like pip found it but decided to discard it for some reason, maybe you should ask pypa/pip. — Reply to this email directly, view it on GitHub <#1680 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABEZV2BQQ22U4H7OOPC6DL3XNGJFFANCNFSM6AAAAAAZSSOTQI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[chrisjsewell]

Have you tried (a) pip install --upgrade pip, to make sure the issue is still there with the latest version of pip, and if so, (b) pip install pip==xxx, to see if you can find out if this works in older pip versions (and what version broke it)

[chrisjsewell]

I tried pip install --upgrade pip and it failed
Then I tried down to pip install --upgrade "pip<20", and it finally started working again 🎉

[chrisjsewell]

Opened pypa/pip#12110

[ChristopherRabotin]

The only solution I have found is to change the version of the crate to one that does not exist on pypi. For example, since I'm developing version 2.0.0-beta, I have changed the crate version to be 2.0.0-beta-dev.

Then, it works.

total 125696
-rw-r--r--  1 runner  staff    30M Jun 26 21:44 nyx_space-2.0.0b0.dev0-cp310-cp310-macosx_10_7_x86_64.whl
-rw-r--r--  1 runner  staff    30M Jun 26 21:46 nyx_space-2.0.0b0.dev0-cp311-cp311-macosx_10_7_x86_64.whl
Looking in links: dist
Processing ./dist/nyx_space-2.0.0b0.dev0-cp311-cp311-macosx_10_7_x86_64.whl
Collecting pandas~=2.0.0 (from nyx_space)
  Downloading pandas-2.0.2-cp311-cp311-macosx_10_9_x86_64.whl (11.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.6/11.6 MB 40.8 MB/s eta 0:00:00