Publish wheel manylinux2014_x86_64 instead of manylinux2014_i686 #1202

fxpineau · 2022-10-17T23:31:24Z

fxpineau
Oct 17, 2022

First of all, thank you for this wonderful tool :)

The github workflow I was using seems to be broken.
I am using the container quay.io/pypa/manylinux2014_i686, install rust, maturin and exec:
maturin publish --compatibility manylinux2014.
The wheel I get is labelled with x86_64 instead of i686.
Here a few details, printing the result of rustc -vV, python and maturin versions:

maturin 0.13.6
rustc 1.64.0 (a55dd71d5 2022-09-19)
binary: rustc
commit-hash: xxx
commit-date: 2022-09-19
host: i686-unknown-linux-gnu
release: 1.64.0
LLVM version: 14.0.6
/opt/python/cp38-cp38/bin/python
📦 Including license file "LICENSE"
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings
🐍 Found CPython 3.8 at /opt/python/cp38-cp38/bin/python
   Compiling pyo3-build-config v0.17.2
   Compiling pyo3-ffi v0.17.2
   Compiling pyo3 v0.17.2
   Compiling numpy v0.17.2
   Compiling xx
    Finished release [optimized] target(s) in 33.06s
📦 Built wheel for CPython 3.8 to /__w/xx/xx/target/wheels/xx-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Am I missing something obvious?

Answered by messense

Oct 19, 2022

Instead of container: quay.io/pypa/manylinux2014_i686 can you try run it via docker run in GitHub Actions?

That image can't even use actions/checkout@v2, its usage is really limited on GitHub Actions via container.

View full answer

messense · 2022-10-18T02:26:00Z

messense
Oct 18, 2022
Maintainer

[root@50f0fba1c908 ~]# maturin new hello
✔ 🤷 Which kind of bindings to use? · pyo3
  ✨ Done! New project created hello
[root@50f0fba1c908 ~]# cd hello/
[root@50f0fba1c908 hello]# ls
Cargo.toml  pyproject.toml  src
[root@50f0fba1c908 hello]# maturin build -o dist
🔗 Found pyo3 bindings
🐍 Found CPython 3.10 at /opt/python/cp310-cp310/bin/python3
   Compiling target-lexicon v0.12.4
   Compiling autocfg v1.1.0
   Compiling proc-macro2 v1.0.47
   Compiling once_cell v1.15.0
   Compiling quote v1.0.21
   Compiling unicode-ident v1.0.5
   Compiling libc v0.2.135
   Compiling syn v1.0.102
   Compiling parking_lot_core v0.9.3
   Compiling scopeguard v1.1.0
   Compiling cfg-if v1.0.0
   Compiling smallvec v1.10.0
   Compiling unindent v0.1.10
   Compiling indoc v1.0.7
   Compiling lock_api v0.4.9
   Compiling memoffset v0.6.5
   Compiling pyo3-build-config v0.17.2
   Compiling parking_lot v0.12.1
   Compiling pyo3-ffi v0.17.2
   Compiling pyo3 v0.17.2
   Compiling pyo3-macros-backend v0.17.2
   Compiling pyo3-macros v0.17.2
   Compiling hello v0.1.0 (/root/hello)
    Finished dev [unoptimized + debuginfo] target(s) in 47.54s
📦 Built wheel for CPython 3.10 to dist/hello-0.1.0-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.whl
[root@50f0fba1c908 hello]# maturin publish
🔗 Found pyo3 bindings
🐍 Found CPython 3.10 at /opt/python/cp310-cp310/bin/python3
   Compiling target-lexicon v0.12.4
   Compiling autocfg v1.1.0
   Compiling once_cell v1.15.0
   Compiling proc-macro2 v1.0.47
   Compiling unicode-ident v1.0.5
   Compiling libc v0.2.135
   Compiling quote v1.0.21
   Compiling syn v1.0.102
   Compiling parking_lot_core v0.9.3
   Compiling scopeguard v1.1.0
   Compiling cfg-if v1.0.0
   Compiling smallvec v1.10.0
   Compiling indoc v1.0.7
   Compiling unindent v0.1.10
   Compiling lock_api v0.4.9
   Compiling memoffset v0.6.5
   Compiling pyo3-build-config v0.17.2
   Compiling parking_lot v0.12.1
   Compiling pyo3-ffi v0.17.2
   Compiling pyo3 v0.17.2
   Compiling pyo3-macros-backend v0.17.2
   Compiling pyo3-macros v0.17.2
   Compiling hello v0.1.0 (/root/hello)
    Finished release [optimized] target(s) in 46.10s
📦 Built wheel for CPython 3.10 to /root/hello/target/wheels/hello-0.1.0-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.whl
📦 Built source distribution to /root/hello/target/wheels/hello-0.1.0.tar.gz

I can't reproduce it locally, same for Python 3.8

[root@50f0fba1c908 hello]# maturin build -o dist
🔗 Found pyo3 bindings
🐍 Found CPython 3.8 at /opt/python/cp38-cp38/bin/python3
   Compiling pyo3-build-config v0.17.2
   Compiling pyo3-ffi v0.17.2
   Compiling pyo3 v0.17.2
   Compiling hello v0.1.0 (/root/hello)
    Finished dev [unoptimized + debuginfo] target(s) in 17.03s
📦 Built wheel for CPython 3.8 to dist/hello-0.1.0-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl

Do you have any Cargo related env vars set, for example CARGO_BUILD_TARGET?

0 replies

fxpineau · 2022-10-18T07:43:39Z

fxpineau
Oct 18, 2022
Author

Thank you very much for your quick answer.

Empty CARGO_BUILD_TARGET.
The main difference with the template we get using maturin new hello (with maturin 13.2) is that I am using pyo3 v17.2 instead of v16.5.
I added --target i686-unknown-linux-gnu to the maturin command line. Same result.
(It worked as expected 11 months ago with a previous version of maturin).

Here the github action CI:


on: [push, workflow_dispatch]

jobs:

  build-linux32-wheels:
    runs-on: ubuntu-latest
    # Docker Hub image that 'build-linux-wheels' executes in.
    # See https://github.com/pypa/manylinux for this particular container:
    # * CPython 3.8, 3.9, ... installed in /opt/python/<python tag>-<abi tag>
    container: quay.io/pypa/manylinux2014_i686
    # We are now in CentOS 7 32 bits
    steps:
     - name: "Checkout the full project"
       uses: actions/checkout@v1
     - name: "Install Rust"
       run: |
         curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --default-host i686-unknown-linux-gnu -y
     # Build wheels for all pre-installed python version (in /opt/python/, see docker image comment)
     # installing first maturin and using maturin: https://github.com/PyO3/maturin#pypy
     - name: "Build and publish wheels"
       shell: bash
       env:
        MATURIN_USERNAME: ${{ secrets.xx }}
        MATURIN_PASSWORD: ${{ secrets.xx }}
       run: |
         source $HOME/.cargo/env
         for PYBIN in /opt/python/cp3[8910]*/bin; do
           "${PYBIN}/pip" install --upgrade pip
           "${PYBIN}/pip" install maturin
           "${PYBIN}/maturin" -V
           rustc -vV
           echo "${PYBIN}/python"
           echo "CARGO_BUILD_TARGET: ${CARGO_BUILD_TARGET}"
           "${PYBIN}/maturin" publish -i "${PYBIN}/python" --skip-existing --compatibility manylinux2014 --username "$MATURIN_USERNAME" --target i686-unknown-linux-gnu
         done

Here the result:

Installing collected packages: maturin
Successfully installed maturin-0.13.6
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
maturin 0.13.6
rustc 1.64.0 (a55dd71d5 2022-09-19)
binary: rustc
commit-hash: a55dd71d5fb0ec5a6a3a9e8c27b2127ba491ce52
commit-date: 2022-09-19
host: i686-unknown-linux-gnu
release: 1.64.0
LLVM version: 14.0.6
/opt/python/cp310-cp310/bin/python
CARGO_BUILD_TARGET: 
📦 Including license file "LICENSE"
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings
📡 Using build options profile, frozen, locked, bindings from pyproject.toml
🐍 Found CPython 3.10 at /opt/python/cp310-cp310/bin/python
   Compiling ...
    ...
    Finished release [optimized] target(s) in 1m 07s
📦 Built wheel for CPython 3.10 to /__w/xx/xx/target/wheels/xx-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

1 reply

messense Oct 18, 2022
Maintainer

Can you reproduce it locally using Docker? Does it only happen on GitHub Actions?

fxpineau · 2022-10-18T11:48:57Z

fxpineau
Oct 18, 2022
Author

I can't reproduce it locally, so it indeed seems to be a Github Actions related issue :o/

0 replies

messense · 2022-10-19T03:45:07Z

messense
Oct 19, 2022
Maintainer

Instead of container: quay.io/pypa/manylinux2014_i686 can you try run it via docker run in GitHub Actions?

That image can't even use actions/checkout@v2, its usage is really limited on GitHub Actions via container.

3 replies

fxpineau Oct 19, 2022
Author

It works as expected with docker run (it do create the i686 wheels).
I guess the current best practice to create python wheels for various architectures is to follow the maturin release script, right?

messense Oct 19, 2022
Maintainer

https://github.com/messense/maturin-action

fxpineau Oct 19, 2022
Author

Thank you very much @messense

fxpineau · 2022-10-19T23:21:12Z

fxpineau
Oct 19, 2022
Author

@messense
I tried to follow your example here: https://github.com/messense/crfs-rs/blob/main/.github/workflows/Python.yml
The linux part (more generally the parts using a docker image) does not work properly with actions/setup-python@v2.
The docker has no access to the python installed with actions/setup-python@v2.
In your run here there is no need for python, but it would not be able to use python 3.9.

2 replies

fxpineau Nov 15, 2022
Author

(@messense, sorry to come back to this, do you agree with my previous message or am I wrong? In any case, thank you again for the work you are doing)

messense Nov 15, 2022
Maintainer

Yes, the docker container only share the git checkout, anything else on the runner isn't accessible.

Python installed via actions/setup-python may have higher glibc version than docker container so it's not safe to mount them into docker container.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Publish wheel manylinux2014_x86_64 instead of manylinux2014_i686 #1202

{{title}}

Replies: 5 comments 6 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Publish wheel manylinux2014_x86_64 instead of manylinux2014_i686 #1202

fxpineau Oct 17, 2022

Replies: 5 comments · 6 replies

messense Oct 18, 2022 Maintainer

fxpineau Oct 18, 2022 Author

messense Oct 18, 2022 Maintainer

fxpineau Oct 18, 2022 Author

messense Oct 19, 2022 Maintainer

fxpineau Oct 19, 2022 Author

messense Oct 19, 2022 Maintainer

fxpineau Oct 19, 2022 Author

fxpineau Oct 19, 2022 Author

fxpineau Nov 15, 2022 Author

messense Nov 15, 2022 Maintainer

fxpineau
Oct 17, 2022

Replies: 5 comments 6 replies

messense
Oct 18, 2022
Maintainer

fxpineau
Oct 18, 2022
Author

messense Oct 18, 2022
Maintainer

fxpineau
Oct 18, 2022
Author

messense
Oct 19, 2022
Maintainer

fxpineau Oct 19, 2022
Author

messense Oct 19, 2022
Maintainer

fxpineau Oct 19, 2022
Author

fxpineau
Oct 19, 2022
Author

fxpineau Nov 15, 2022
Author

messense Nov 15, 2022
Maintainer