How to specify the install path for DLLs?

[polm]

I have a package I distribute that has wheels for OSX, Linux, and Windows. On Windows I include a DLL file. I've always specified this in data_files in the call to setup(), which I copied from an older package I took over maintenance of. This has always worked when I've tested it on Windows. Apparently it's wrong though. What's the right thing?

Occasionally I get reports from Windows users that their package was installed without a DLL somehow. There's no clear pattern to these users, or none that I can recognize at least. I learned that the way I was specifying things was wrong when I asked what was going on at the pip repo - they say I should use platlib. I am not sure how platlib relates to setuptools, and neither were the pip devs.

Note I don't really know anything about Windows programming and only do it as part of supporting this or other packages I maintain.

[webknjaz]

It'd be better if you linked the source code of your project. Or some examples, at least. I'm not a windows person myself but I'd like to point out that people normally don't include pre-existing C-extensions into wheels but compile them when building wheels from the source. Also, if those DLLs are not C-extensions but really some data that your package loads/uses in runtime, maybe use package_data/include_package_data instead of data_files. https://setuptools.pypa.io/en/latest/userguide/quickstart.html#including-data-files

Oh, and look at https://cibuildwheel.rtfd.io that may be helpful to you for building a huge matrix of wheels... I can't offer you a more specific answer because I'm not sure what you're asking exactly.

I've found that the distutils command build vendored in (and inherited by) setuptools has some platlib options but there's no docs about it, plus it'd rely on a deprecated invocation style (pre-PEP517 one).
I seem to recall, though, that setuptools/distutils automatically detect type of the content and mark things as platlib or purelib internally automatically, based on the presence of C-extensions in the project.

[polm]

Sorry for not including more details, but they're at the issue linked above. The package I'm referring to is fugashi. It's a wrapper for a C++ library that's hard to build on Windows and not actively maintained, which is why I include a DLL. The older package I used as a reference is mecab-python3, which I didn't create but do maintain now, which wraps the same C++ library. (The DLL is not just data.)

I don't need help building a matrix of wheels, and I've used cibuildwheel before. This is just a question about the install path for the DLL.

The specific issue is that I want the DLL installed somewhere it can be found at runtime. On my Windows machines, where the install works, the DLL is at site-packages/fugashi, so that the pyd file, other sources, and the dll are together. I'm not sure if they have to be in the same directory (I don't completely understand the Windows DLL lookup process) but if there's a way to specify that directory I'd be happy to use it (and I thought that's what data_files was doing). Is there some way to specify that directory? If not is there a way to specify a directory where I can be sure the DLL will be found at runtime?

I understand that including prebuilt DLLS may not be the most common thing to do, but I am surprised there isn't some typical way to do it.

[mhkline]

What did you end up doing @polm? I'm facing a similar situation. I have a pre-built DLL wrapped with ctypes, and I'd rather not rearchitect everything as a C-extension.

I'm also confused about how to make the wheel platform-specific when the DLL is included as a data file.

[polm]

I have not figured out a solution for this on my end. It seems it only affects a small portion of Windows users.

I'm still confused about the details, but I think it only happens if doing a pip install outside a virtualenv, so using --user or a virtualenv solves the issue.

[Learath]

I can reliably replicate this problem if it would help @polm

PS D:\src\text-generation-webui> python3 -m pip install --user fugashi
Collecting fugashi
Using cached fugashi-1.3.0-cp311-cp311-win_amd64.whl.metadata (7.1 kB)
Using cached fugashi-1.3.0-cp311-cp311-win_amd64.whl (498 kB)
Installing collected packages: fugashi
WARNING: The scripts fugashi-build-dict.exe, fugashi-info.exe and fugashi.exe are installed in 'C:\Users\Nick\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\Scripts' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed fugashi-1.3.0
PS D:\src\text-generation-webui> python -m fugashi
C:\Users\Nick\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\python.exe: Error while finding module specification for 'fugashi.main' (ImportError: DLL load failed while importing fugashi: The specified module could not be found.)
PS D:\src\text-generation-webui>

[polm]

Thanks, I would appreciate a report. Since this is the pypa/setuptools repo, we can follow up on the details at the fugashi repo. If you could open an issue at the fugashi repo and also do the following:

• mention how you installed Python
• use the fugashi script (do not use python -m fugashi, it is not designed to work)
• link to this discussion

Also please use markdown formatting for any code or output you paste.

[Learath]

I've solved it for me at least, I have to manually add the dll path.

 os.add_dll_directory("C:\\Users\\<my user>\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\\LocalCache\\local-packages\\lib\\site-packages\\fugashi")

[jpshivakavi]

You could do the same thing programmatically as below:
import os.path
import sys
dll_path=os.path.join(os.path.dirname(sys.executable), 'Lib', 'site-packages', 'fugashi') # this makes it work for other versions of Python3 as well
os.add_dll_directory(dll_path)