simple index suddenly stopped working for scipy-openblas32, scipy-openblas64

[mattip]

NumPy has a CI job that tries to pip install -i https://pypi.anaconda.org/scientific-python-nightly-wheels/simple scipy-openblas32. It started failing 16 hours ago, which is similar to the time of the last upload to the archive site. Note the files on the site are named scipy-openblas32 with a -. I think pip is expecting files with a _ like on PyPI. But I could not upload files with a _ in the filename. When using twine, the rename is done by the uploader. I opened an issue with anaconda-client anaconda/anaconda-client#704.

I am not sure what changed with the last upload: did it suddenly rename all the files? In any case, perhaps it would be good to rename the project to scipy_openblas32 and scipy_openblas64 to avoid fighting these problems. I wonder how scikit-image is uploading files with the _?

[matthewfeickert]

I am not sure what changed with the last upload: did it suddenly rename all the files?

@mattip Looking at the files https://anaconda.org/scientific-python-nightly-wheels/scipy-openblas32/files there's been no change to the way the files you've been uploading have been named in at least the last 13 days.

While I agree that installing from the index fails

$ docker run --rm -ti python:3.11 /bin/bash
root@ac06741d6c27:/# python -m venv venv && . venv/bin/activate
(venv) root@ac06741d6c27:/# python -m pip --quiet install --upgrade pip setuptools wheel
(venv) root@ac06741d6c27:/# python -m pip install -i https://pypi.anaconda.org/scientific-python-nightly-wheels/simple scipy-openblas32
Looking in indexes: https://pypi.anaconda.org/scientific-python-nightly-wheels/simple
ERROR: Could not find a version that satisfies the requirement scipy-openblas32 (from versions: none)
ERROR: No matching distribution found for scipy-openblas32
(venv) root@ac06741d6c27:/# 

I'm not sure if there's something wrong with the index or if the issue is with the wheel metadata given that the file scipy-openblas32-0.3.26.199-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl exists and

python -m pip install \
  --upgrade \
  --pre \
  --index-url https://pypi.anaconda.org/scientific-python-nightly-wheels/simple \
  --extra-index-url https://pypi.org/simple \
  scikit-image

works.

As this isn't something that is related to the upload-nightly-action I would suggest that you continue to document the problems you're having on anaconda/anaconda-client#704 that you already opened. Though we're of course happy to debug with you along the way.

[matthewfeickert]

In any case, perhaps it would be good to rename the project to scipy_openblas32 and scipy_openblas64 to avoid fighting these problems. I wonder how scikit-image is uploading files with the _?

I'd suggest asking @jarrodmillman or @lagru given the history of their wheel building CI https://github.com/scikit-image/scikit-image/commits/main/.github/workflows/wheels_recipe.yml. (I think this is metadata related and not uploader related, so I'd start looking at the wheels first.)

[matthewfeickert]

For some additional information, on 2024-02-17 at 20:01 CST the remove old wheels workflow ran and removed the v0.3.26.0 versions of scipy-openblas32 and scipy-openblas64

# package: ipython

# package: matplotlib

# package: networkx

# package: numpy

# package: pandas
# Removing scientific-python-nightly-wheels/pandas/3.0.0.dev0+301.g4e40afd9d9
Using Anaconda API: https://api.anaconda.org

# package: scikit-image

# package: scikit-learn

# package: scipy

# package: scipy-openblas32
# Removing scientific-python-nightly-wheels/scipy-openblas32/0.3.26.0
Using Anaconda API: https://api.anaconda.org

# package: scipy-openblas[64](https://github.com/scientific-python/upload-nightly-action/actions/runs/7945846045/job/21692939840#step:6:65)
# Removing scientific-python-nightly-wheels/scipy-openblas64/0.3.26.0
Using Anaconda API: https://api.anaconda.org/

# package: statsmodels

# package: test-package

# package: xarray

in keeping with the 30 day retention policy

upload-nightly-action/README.md

Lines 69 to 73 in 66bc1b6

	To avoid hosting outdated development versions, as well as to clean up space, we do have a
	default retention policy of:
	- Latest **5 versions**
	- Artifacts newer than **30 days**

On 2024-02-18 at 11:22 CST your CI job ran and failed

Looking in indexes: https://pypi.anaconda.org/scientific-python-nightly-wheels/simple
ERROR: Could not find a version that satisfies the requirement scipy-openblas32 (from versions: none)
ERROR: No matching distribution found for scipy-openblas32

Given that there have been no changes to the naming scheme for these scipy-openblas32/64 wheels for at least 2 weeks, my hunch is that 0.3.26.0 was the last valid wheel that had been uploaded and so that wheel has been getting downloaded until it got removed. My hunch is correct, as looking at the last worklow that ran succesfully (2024-02-17 at 19:26 CST) for numpy/numpy#19753 (https://github.com/numpy/numpy/actions/runs/7945606976/job/21692459207) we can see that wheel it downloads is 0.3.26.0

Looking in indexes: https://pypi.anaconda.org/scientific-python-nightly-wheels/simple
Collecting scipy-openblas32
  Downloading https://pypi.anaconda.org/scientific-python-nightly-wheels/simple/scipy-openblas32/0.3.26.0/scipy_openblas32-0.3.26.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.3/11.3 MB 32.0 MB/s eta 0:00:00
Installing collected packages: scipy-openblas32
Successfully installed scipy-openblas32-0.3.26.0

@mattip @rgommers As this is definitley a problem with the wheel metadata, have there been any changes to your wheel building procedure in the last 30 days?

[matthewfeickert]

Hm. @mattip While I would like to think that pip would be robust to names here, it does seem like that this is a problem of the - vs. _ in the filename given the following example.

$ docker run --rm -ti python:3.11 /bin/bash
root@c27a965fe2ac:/# python -m venv venv && . venv/bin/activate
(venv) root@c27a965fe2ac:/# python -m pip install --quiet --upgrade pip wheel
(venv) root@c27a965fe2ac:/# curl -sLO https://anaconda.org/scientific-python-nightly-wheels/scipy-openblas32/0.3.26.199/download/scipy-openblas32-0.3.26.199-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl  # manually download the wheel file
(venv) root@c27a965fe2ac:/# python -m pip install --quiet twine
(venv) root@c27a965fe2ac:/# twine check --strict scipy-openblas32*.whl 
Checking scipy-openblas32-0.3.26.199-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl: PASSED
(venv) root@c27a965fe2ac:/# python -m pip install --upgrade ./scipy-openblas32-0.3.26.199-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 
Processing /scipy-openblas32-0.3.26.199-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Discarding file:///scipy-openblas32-0.3.26.199-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl: Requested scipy==openblas32 from file:///scipy-openblas32-0.3.26.199-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl has inconsistent name: expected 'scipy', but metadata has 'scipy-openblas32'
ERROR: Ignored the following yanked versions: 1.11.0
ERROR: Ignored the following versions that require a different python version: 1.6.2 Requires-Python >=3.7,<3.10; 1.6.3 Requires-Python >=3.7,<3.10; 1.7.0 Requires-Python >=3.7,<3.10; 1.7.1 Requires-Python >=3.7,<3.10; 1.7.2 Requires-Python >=3.7,<3.11; 1.7.3 Requires-Python >=3.7,<3.11; 1.8.0 Requires-Python >=3.8,<3.11; 1.8.0rc1 Requires-Python >=3.8,<3.11; 1.8.0rc2 Requires-Python >=3.8,<3.11; 1.8.0rc3 Requires-Python >=3.8,<3.11; 1.8.0rc4 Requires-Python >=3.8,<3.11; 1.8.1 Requires-Python >=3.8,<3.11
ERROR: Could not find a version that satisfies the requirement scipy (unavailable) (from versions: 0.8.0, 0.9.0, 0.10.0, 0.10.1, 0.11.0, 0.12.0, 0.12.1, 0.13.0, 0.13.1, 0.13.2, 0.13.3, 0.14.0, 0.14.1, 0.15.0, 0.15.1, 0.16.0, 0.16.1, 0.17.0, 0.17.1, 0.18.0, 0.18.1, 0.19.0, 0.19.1, 1.0.0, 1.0.1, 1.1.0, 1.2.0, 1.2.1, 1.2.2, 1.2.3, 1.3.0, 1.3.1, 1.3.2, 1.3.3, 1.4.0, 1.4.1, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.6.0, 1.6.1, 1.9.0rc1, 1.9.0rc2, 1.9.0rc3, 1.9.0, 1.9.1, 1.9.2, 1.9.3, 1.10.0rc1, 1.10.0rc2, 1.10.0, 1.10.1, 1.11.0rc1, 1.11.0rc2, 1.11.1, 1.11.2, 1.11.3, 1.11.4, 1.12.0rc1, 1.12.0rc2, 1.12.0)
ERROR: No matching distribution found for scipy (unavailable)
(venv) root@c27a965fe2ac:/# mv scipy-openblas32-0.3.26.199-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl scipy_openblas32-0.3.26.199-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
(venv) root@c27a965fe2ac:/# python -m pip install --upgrade ./scipy_openblas32-0.3.26.199-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl  # rename to the original wheel name when built
Processing /scipy_openblas32-0.3.26.199-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Installing collected packages: scipy-openblas32
Successfully installed scipy-openblas32-0.3.26.199
(venv) root@c27a965fe2ac:/#

I wonder if this is due to the fact that in MacPython/openblas-libs#135 you're manually renaming the wheel after having built and verified it.

# from your build logs
 for f in dist/*.whl
+ cp dist/scipy_openblas64-0.3.26.199-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl dist/scipy-openblas64-0.3.26.199-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

I would not do this. There is definitley no need to at least.

[mattip]

Thanks for the analysis, I guess MacPython/openblas-libs#135 never worked properly (?) even though I was pretty sure I checked it at the time. All this is quite frustrating, the twine uploader together with PyPI has no problem handling these wheels.

This is not really a problem with the upload-nightly-action, so further action should happen in other places: maybe anaconda/anaconda-client#704, maybe MacPython/openblas-libs#139.

There is definitely no need to at least.

I wouldn't have done that PR if there was no need.

[matthewfeickert]

@mattip I've tried to answer your question over on MacPython/openblas-libs#140 (comment).

All this is quite frustrating, the twine uploader together with PyPI has no problem handling these wheels.

While I show that this is not uploader related at all, can you plink to an example of where you're uploading wheels with this naming scheme to PyPI with twine?

I wouldn't have done that PR if there was no need.

MacPython/openblas-libs#135 was necessary to change the project name but you also changed the wheel filename which was not needed and is what is causing the problem.

[mattip]

Could someone with admin priveleges delete all the wheels at https://anaconda.org/scientific-python-nightly-wheels/scipy-openblas32/files older than 1 day? Something is off with that package site, and perhaps somehow the broken wheels (hopefully fixed by MacPython/openblas-libs#140) are interfering with the indexing?

[rgommers]

Still necessary, or is it working now (MacPython/openblas-libs#139 (comment) is more recent)?

[mattip]

Deleting wheels is not needed. Thanks to @matthewfeickert the problem is solved.

[rgommers]

phew 😓. thanks for the hard work to both of you!

[matthewfeickert]

Could someone with admin priveleges delete all the wheels at https://anaconda.org/scientific-python-nightly-wheels/scipy-openblas32/files older than 1 day?

Done. I left the stable/release version of 0.3.26.0.5, but beyond that it is just the 1 nightly from @mattip's triggered rebuild post MacPython/openblas-libs#140: 0.3.26.201. So should be good now!

Thaks very much @mattip for working through this with me! While I know this was a bit of a pain for the NumPy/SciPy/OpenBLAS teams, I think I now have a much better handle on how all the nightlies get built after this little adventure, so I hope that I can be more helpful to you all in the future if there are other issues related to the https://anaconda.org/scientific-python-nightly-wheels/ index!