Release is broken: delocate tool finds inconsistencies in packages under macos-13 and macos-14 GitHub action environments

[vsbogd]

Describe the bug
After upgrade to cibuildwheel 2.19.2 (see PR #724) version 0.11.0 of the delocate library is used while releasing binaries. This version does additional consistency checks on the Python packages built. It finds out that libssl and libcrypto libraries used are targeted for the MacOS 13 while package is built for the MacOS 10.9 which breaks the release of the package on macos-13 GitHub actions environment. It also fails finding correct minimum MacOS version for the arm64 platform on macos-14 GitHub actions environment.

Full log at https://github.com/trueagi-io/hyperon-experimental/actions/runs/9797477257

To Reproduce
Steps to reproduce the behavior:

1. Make test branch in your fork from the following commit 995fa92
2. Start test release of the branch in your fork see https://github.com/trueagi-io/hyperon-experimental/blob/main/docs/DEVELOPMENT.md#how-to-check-release-job-in-fork

Expected behavior
Python packages for the macos-13 and macos-14 are released

Actual behavior
Release fails on macos-13 with the following delocate error message:

2024-07-04T16:31:09.3113540Z delocate.libsana.DelocationError: Library dependencies do not satisfy target MacOS version 10.9:
2024-07-04T16:31:09.3115990Z /private/var/folders/ng/t2lj_z6n4h30fn0h2ql_nd0c0000gn/T/tmph3qhc4hr/wheel/hyperon/.dylibs/libcrypto.3.dylib has a minimum target of 13.0
2024-07-04T16:31:09.3119610Z /private/var/folders/ng/t2lj_z6n4h30fn0h2ql_nd0c0000gn/T/tmph3qhc4hr/wheel/hyperon/.dylibs/libssl.3.dylib has a minimum target of 13.0

Release fails on macos-14 with the following delocate error message:

2024-07-04T16:26:54.4328210Z + delocate-wheel --require-archs arm64 -w /private/var/folders/h9/l1shxhdd69nct08ylzq0n6q00000gn/T/cibw-run-34b9atwn/cp38-macosx_arm64/repaired_wheel -v /private/var/folders/h9/l1shxhdd69nct08ylzq0n6q00000gn/T/cibw-run-34b9atwn/cp38-macosx_arm64/built_wheel/hyperon-0.1.11-cp38-cp38-macosx_11_0_arm64.whl
2024-07-04T16:26:55.0927940Z Fixing: /private/var/folders/h9/l1shxhdd69nct08ylzq0n6q00000gn/T/cibw-run-34b9atwn/cp38-macosx_arm64/built_wheel/hyperon-0.1.11-cp38-cp38-macosx_11_0_arm64.whl
2024-07-04T16:26:55.0988460Z Traceback (most recent call last):
2024-07-04T16:26:55.0997840Z   File "/private/var/folders/h9/l1shxhdd69nct08ylzq0n6q00000gn/T/cibw-run-34b9atwn/cp38-macosx_arm64/build/venv/lib/python3.8/site-packages/delocate/delocating.py", line 789, in _calculate_minimum_wheel_name
2024-07-04T16:26:55.0999020Z     arch_version[arch] = max(version, version_dkt[arch])
2024-07-04T16:26:55.0999420Z KeyError: 'arm64'
...
2024-07-04T16:26:55.1076450Z delocate.libsana.DelocationError: Failed to find any binary with the required architecture: 'arm64'

Additional context
It should be noted that libssl and libcrypto shared libraries are included into the packages.
See also https://cibuildwheel.pypa.io/en/stable/faq/#macos-library-dependencies-do-not-satisfy-target-macos

[vsbogd]

I am tending to think delocate behaviour on macos-14 is a bug as I manually checked artifacts and I see libraries inside are arm64 ones.

[vsbogd]

macos-13 behaviour also looks logical as system versions of libssl and libcrypto are used and as system version is 13 the libraries version is 13 as well. I am not sure who is responsible for targeting to the proper libraries:

1. where to get libraries for 10.9 inside cibuildwheel environment?
2. is it possible to set artifacts minimal version to 13 using cibuildwheel?

[vegardjervell]

I came across this when I was having a similar problem myself. The quick-fix I found to ensure that MACOSX_DEPLOYMENT_TARGET is set to the minimal version of the libs installed on the runner was adding the step

- name: Set macOS deployment target
  run: echo "MACOSX_DEPLOYMENT_TARGET=$(sw_vers -productVersion | cut -d '.' -f 1-2)" >> $GITHUB_ENV

before wheel or delocate are called. This doesn't ensure that your libs have a particular version, but at least ensures that wheel and delocate are on the same page regarding what the target version should be (the version of the runner). If you want to target versions lower than the version of the runner I think you could get that done by specifying the version the same way, and then installing the proper versions of the libs with brew before building the project.

As far as I'm aware, GitHub hosted runners only come with brew's default installation of libraries for the runner, which target the specific version of the runner, not any lower versions.

[vsbogd]

@vegardjervell thank you for the tip. It fixes issue with delocate but unfortunately it breaks package testing under cibuildwheel. Setting it to the OS version (13.6 for macos-13, 14.5 for macos-14) allows building package successfully but testing fails as far as I see because testing environment is 10.9 and it claims 13.6 or 14.5 packages are not supported.

See https://github.com/vsbogd/hyperon-experimental/actions/runs/10176752594

...
2024-07-31T08:26:55.1077260Z   environment: 
2024-07-31T08:26:55.1078680Z     MACOSX_DEPLOYMENT_TARGET=13.6ESC[0m
...
2024-07-31T08:34:43.7380700Z + delocate-wheel --require-archs x86_64 -w /private/var/folders/z0/8_glkdsj15s73lhzlcw256wm0000gn/T/cibw-run-s7ubqtgo/cp37-macosx_x86_64/repaired_wheel 
-v /private/var/folders/z0/8_glkdsj15s73lhzlcw256wm0000gn/T/cibw-run-s7ubqtgo/cp37-macosx_x86_64/built_wheel/hyperon-0.2.0-cp37-cp37m-macosx_13_0_x86_64.whl
2024-07-31T08:34:45.4669840Z INFO:delocate.delocating:Copying library /usr/local/Cellar/openssl@3/3.3.1/lib/libssl.3.dylib to hyperon/.dylibs/libssl.3.dylib
2024-07-31T08:34:45.4747040Z INFO:delocate.delocating:Copying library /usr/local/Cellar/openssl@3/3.3.1/lib/libcrypto.3.dylib to hyperon/.dylibs/libcrypto.3.dylib
2024-07-31T08:34:45.4874500Z INFO:delocate.delocating:Modifying install name in hyperonpy.cpython-37m-darwin.so from /usr/local/opt/openssl@3/lib/libssl.3.dylib to @loader_path/hype
ron/.dylibs/libssl.3.dylib
2024-07-31T08:34:46.5544600Z INFO:delocate.delocating:Modifying install name in hyperonpy.cpython-37m-darwin.so from /usr/local/opt/openssl@3/lib/libcrypto.3.dylib to @loader_path/h
yperon/.dylibs/libcrypto.3.dylib
2024-07-31T08:34:46.6696940Z INFO:delocate.delocating:Modifying install name in hyperon/.dylibs/libssl.3.dylib from /usr/local/Cellar/openssl@3/3.3.1/lib/libcrypto.3.dylib to @loade
r_path/libcrypto.3.dylib
2024-07-31T08:34:48.5548010Z Fixing: /private/var/folders/z0/8_glkdsj15s73lhzlcw256wm0000gn/T/cibw-run-s7ubqtgo/cp37-macosx_x86_64/built_wheel/hyperon-0.2.0-cp37-cp37m-macosx_13_0_x86_64.whl
2024-07-31T08:34:48.5551560Z Copied to package .dylibs directory:
2024-07-31T08:34:48.5554010Z   /usr/local/Cellar/openssl@3/3.3.1/lib/libcrypto.3.dylib
2024-07-31T08:34:48.5556720Z   /usr/local/Cellar/openssl@3/3.3.1/lib/libssl.3.dylib
2024-07-31T08:34:48.5641210Z ##[endgroup]
...
2024-07-31T08:34:52.2664670Z created virtual environment CPython3.7.9.final.0-64 in 1075ms
2024-07-31T08:34:52.2666970Z   creator CPython3Posix(dest=/private/var/folders/z0/8_glkdsj15s73lhzlcw256wm0000gn/T/cibw-run-s7ubqtgo/cp37-macosx_x86_64/venv-test-x86_64, clear=False, no_vcs_ignore=False, global=False)
2024-07-31T08:34:52.2669590Z   seeder FromAppData(download=False, pip=24.0, setuptools=embed, wheel=embed, via=copy, app_data_dir=/Users/runner/Library/Application Support/virtualenv)
2024-07-31T08:34:52.2671240Z     added seed packages: pip==24.0, setuptools==68.0.0, wheel==0.42.0
2024-07-31T08:34:52.2672680Z   activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
2024-07-31T08:34:52.2890250Z + which python
2024-07-31T08:34:52.2937740Z /private/var/folders/z0/8_glkdsj15s73lhzlcw256wm0000gn/T/cibw-run-s7ubqtgo/cp37-macosx_x86_64/venv-test-x86_64/bin/python
2024-07-31T08:34:52.2941770Z + pip install /private/var/folders/z0/8_glkdsj15s73lhzlcw256wm0000gn/T/cibw-run-s7ubqtgo/cp37-macosx_x86_64/repaired_wheel/hyperon-0.2.0-cp37-cp37m-macosx_13_6_x86_64.whl
2024-07-31T08:34:53.4978390Z ERROR: hyperon-0.2.0-cp37-cp37m-macosx_13_6_x86_64.whl is not a supported wheel on this platform.
...