How to repair path to macOS Python in compiled wheel? #991

DBraun · 2022-01-11T16:06:45Z

DBraun
Jan 11, 2022

I'm compiling dawdreamer.so as a custom python library and putting it inside a directory which gets built as a wheel.
dawdreamer:

__init__.py
dawdreamer.so
libfaust.2.dylib

Everything works on Windows and macOS locally, but I'm still having trouble with testing the wheel on macOS after repairing it with cibuildwheel. In CIBW_BEFORE_BUILD, this is the output of otool -L dawdreamer.so:

dawdreamer.so (architecture x86_64):
	/usr/local/lib/dawdreamer.so.dylib (compatibility version 1.0.0, current version 1.0.0)
	@loader_path/libfaust.2.dylib (compatibility version 2.0.0, current version 2.38.7)
	/Library/Frameworks/Python.framework/Versions/3.9/Python (compatibility version 3.9.0, current version 3.9.0)
	/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)
	/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox (compatibility version 1.0.0, current version 1000.0.0)
	/System/Library/Frameworks/AVFoundation.framework/Versions/A/AVFoundation (compatibility version 1.0.0, current version 2.0.0)
	/System/Library/Frameworks/AVKit.framework/Versions/A/AVKit (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon (compatibility version 2.0.0, current version 165.0.0)
	/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 23.0.0)
	/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/CoreAudioKit.framework/Versions/A/CoreAudioKit (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/CoreMedia.framework/Versions/A/CoreMedia (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/CoreMIDI.framework/Versions/A/CoreMIDI (compatibility version 1.0.0, current version 69.0.0)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1856.105.0)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore (compatibility version 1.2.0, current version 1.11.0)
	/System/Library/Frameworks/WebKit.framework/Versions/A/WebKit (compatibility version 1.0.0, current version 612.3.6)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1200.3.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
	/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 2113.20.111)
	/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 56.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1856.105.0)
	/System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics (compatibility version 64.0.0, current version 1557.3.2)
	/System/Library/Frameworks/CoreImage.framework/Versions/A/CoreImage (compatibility version 1.0.1, current version 5.0.0)
	/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 1141.1.0)
	/System/Library/Frameworks/CoreText.framework/Versions/A/CoreText (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/CoreVideo.framework/Versions/A/CoreVideo (compatibility version 1.2.0, current version 1.5.0)
	/System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
dawdreamer.so (architecture arm64):
	/usr/local/lib/dawdreamer.so.dylib (compatibility version 1.0.0, current version 1.0.0)
	@loader_path/libfaust.2.dylib (compatibility version 2.0.0, current version 2.38.7)
	/Library/Frameworks/Python.framework/Versions/3.9/Python (compatibility version 3.9.0, current version 3.9.0)
	/System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)
	/System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox (compatibility version 1.0.0, current version 1000.0.0)
	/System/Library/Frameworks/AVFoundation.framework/Versions/A/AVFoundation (compatibility version 1.0.0, current version 2.0.0)
	/System/Library/Frameworks/AVKit.framework/Versions/A/AVKit (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon (compatibility version 2.0.0, current version 165.0.0)
	/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 23.0.0)
	/System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/CoreAudioKit.framework/Versions/A/CoreAudioKit (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/CoreMedia.framework/Versions/A/CoreMedia (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/CoreMIDI.framework/Versions/A/CoreMIDI (compatibility version 1.0.0, current version 69.0.0)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1856.105.0)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore (compatibility version 1.2.0, current version 1.11.0)
	/System/Library/Frameworks/WebKit.framework/Versions/A/WebKit (compatibility version 1.0.0, current version 612.3.6)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1200.3.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
	/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 2113.20.111)
	/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 56.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1856.105.0)
	/System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics (compatibility version 64.0.0, current version 1557.3.2)
	/System/Library/Frameworks/CoreImage.framework/Versions/A/CoreImage (compatibility version 1.0.1, current version 5.0.0)
	/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 1141.1.0)
	/System/Library/Frameworks/CoreText.framework/Versions/A/CoreText (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/CoreVideo.framework/Versions/A/CoreVideo (compatibility version 1.2.0, current version 1.5.0)
	/System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO (compatibility version 1.0.0, current version 1.0.0)

My question is should I use install_name_tool -change to change /Library/Frameworks/Python.framework/Versions/3.9/Python before building and repairing the wheel?

The following is with:

CIBW_ARCHS: auto64
CIBW_ARCHS_MACOS: x86_64 universal2 
CIBW_BUILD: cp38-macosx_x86_64

I don't specify CIBW_REPAIR_WHEEL_COMMAND and so this is the output of repairing the wheel

  + delocate-listdeps /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibuildwheellt977_xm/built_wheel/dawdreamer-0.5.8.1-cp38-cp38-macosx_10_15_x86_64.whl && delocate-wheel --require-archs x86_64 -w /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibuildwheellt977_xm/repaired_wheel /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibuildwheellt977_xm/built_wheel/dawdreamer-0.5.8.1-cp38-cp38-macosx_10_15_x86_64.whl
  /Library/Frameworks/Python.framework/Versions/3.8/Python
  dawdreamer/libfaust.2.dylib

When pytest runs in the CIBW_TEST_COMMAND, it crashes:

cd /Users/runner/work/DawDreamer/DawDreamer/tests; python -m pytest -s .
  Fatal Python error: PyMUTEX_LOCK(gil->mutex) failed
  Python runtime state: unknown
  
  Fatal Python error: Aborted
  
  Current thread 0x0000000112789dc0 (most recent call first):
    File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
    File "<frozen importlib._bootstrap_external>", line 1166 in create_module
    File "<frozen importlib._bootstrap>", line 556 in module_from_spec
    File "<frozen importlib._bootstrap>", line 657 in _load_unlocked
    File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 991 in _find_and_load
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/dawdreamer/__init__.py", line 8 in <module>
    File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
    File "<frozen importlib._bootstrap_external>", line 848 in exec_module
    File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
    File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 991 in _find_and_load
    File "/Users/runner/work/DawDreamer/DawDreamer/tests/utils.py", line 7 in <module>
    File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
    File "<frozen importlib._bootstrap_external>", line 848 in exec_module
    File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
    File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 991 in _find_and_load
    File "/Users/runner/work/DawDreamer/DawDreamer/tests/test_faust_dx7.py", line 1 in <module>
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/_pytest/assertion/rewrite.py", line 170 in exec_module
    File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
    File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 991 in _find_and_load
    File "<frozen importlib._bootstrap>", line 1014 in _gcd_import
    File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py", line 127 in import_module
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/_pytest/pathlib.py", line 524 in import_path
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/_pytest/python.py", line 578 in _importtestmodule
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/_pytest/python.py", line 500 in _getobj
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/_pytest/python.py", line 291 in obj
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/_pytest/python.py", line 516 in _inject_setup_module_fixture
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/_pytest/python.py", line 503 in collect
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/_pytest/runner.py", line 341 in <lambda>
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/_pytest/runner.py", line 311 in from_call
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/_pytest/runner.py", line 341 in pytest_make_collect_report
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/_pytest/runner.py", line 458 in collect_one_node
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/_pytest/main.py", line 808 in genitems
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/_pytest/main.py", line 634 in perform_collect
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/_pytest/main.py", line 333 in pytest_collection
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/_pytest/main.py", line 322 in _main
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/_pytest/main.py", line 269 in wrap_session
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/_pytest/config/__init__.py", line 162 in main
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/_pytest/config/__init__.py", line 185 in console_main
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpzav2qzk6/lib/python3.8/site-packages/pytest/__main__.py", line 5 in <module>
    File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 87 in _run_code
    File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 194 in _run_module_as_main
  /bin/sh: line 1: 10646 Abort trap: 6           python -m pytest -s .
                                                             ✕ 38.36s
Error: Command  cd /Users/runner/work/DawDreamer/DawDreamer/tests; python -m pytest -s . failed with code 134. None

Error: Process completed with exit code 1.

I'm pretty sure this is due to the path to python not working because the same tests pass after building the wheel locally on macOS/Windows.

Answered by DBraun

Jan 12, 2022

The solution was to not link to Python at all. pybind11 mentions this https://pybind11.readthedocs.io/en/stable/compiling.html

On macOS: the build command is almost the same but it also requires passing the -undefined dynamic_lookup flag so as to ignore missing symbols when building the module

View full answer

Czaki · 2022-01-11T16:31:19Z

Czaki
Jan 11, 2022
Collaborator

Could you explain if you build dawdreamer.so and libfaust.2.dylib before building wheels or during build wheel?

If before you could fix it also before using install_name_tool

example here https://github.com/Czaki/imagecodecs_build/blob/master/build_utils/fix_macos_lib.sh

2 replies

DBraun Jan 11, 2022
Author

dawdreamer.so and libfaust.2.dylib are fully compiled before building the wheel. The directory containing them has a blank file null.c. The setup.py compiles this blank file and also bundles the so and dylib. Also, libfaust.2.dylib has no special dependencies.

My workaround in the past has been to use

install_name_tool -change /Library/Frameworks/Python.framework/Versions/3.9/Python @executable_path/../../../../Python dawdreamer.so

and then build and repair with cibuildwheel. I think this leads to cibuildwheel not changing the path to Python. However, I then must skip the CIBW_TEST_COMMAND. When I download this sort of wheel, it works in a normal Python installation but not a virtual environment. I think this mirrors what's going on with cibuildwheel and why I had to skip the tests.

In a normal Python installation, @executable_path seems to resolve to /Library/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/
Therefore, @executable_path/../../../../Python resolves to /Library/Frameworks/Python.framework/Versions/3.9/Python and the pytest works. However, this trick doesn't work for virtual environments. So I'm confused on how to get a wheel to work in both kinds of situations and test it with cibuildwheel.

Czaki Jan 11, 2022
Collaborator

I do not know. Base on my knowledge any assumption about this path could be bad. I suggest looking in another library and finding one linking to python and checking how it looks. At this moment I do not have access to a mac.

DBraun · 2022-01-12T15:33:04Z

DBraun
Jan 12, 2022
Author

The solution was to not link to Python at all. pybind11 mentions this https://pybind11.readthedocs.io/en/stable/compiling.html

On macOS: the build command is almost the same but it also requires passing the -undefined dynamic_lookup flag so as to ignore missing symbols when building the module

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to repair path to macOS Python in compiled wheel? #991

{{title}}

Replies: 2 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

How to repair path to macOS Python in compiled wheel? #991

DBraun Jan 11, 2022

Replies: 2 comments · 2 replies

Czaki Jan 11, 2022 Collaborator

DBraun Jan 11, 2022 Author

Czaki Jan 11, 2022 Collaborator

DBraun Jan 12, 2022 Author

DBraun
Jan 11, 2022

Replies: 2 comments 2 replies

Czaki
Jan 11, 2022
Collaborator

DBraun Jan 11, 2022
Author

Czaki Jan 11, 2022
Collaborator

DBraun
Jan 12, 2022
Author