-
Notifications
You must be signed in to change notification settings - Fork 236
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Distribute Python sdist
through PyPI
#228
Comments
Hi @diegoferigo, Thanks for the detailed break down of releasing in PyPi. To immediately answer this ticket, yes a PyPI release is on my todo list and I have briefly looked into it at the time of developing #201 (mainly this tutorial). However there are a couple open PRs that I'd like to land before issuing a new tag and releasing to PyPI.
You mean other C++ project right? I had not think of that but I guess it makes sense. |
Great! No problem, do as many attempts as you prefer. One advice I can give you is to always use the testing index test.pypi.org instead of the official one during your early experiment. In fact, if you create a new package in the official index and push by mistake a wrong version, you are not allowed to override it. You can delete it, but then you have lost the usage of that version if needed in the future. (I personally did this mistake in the beginning). As the blog post you linked reports, manually uploading with twine is the most simple start, that could be automatize with CI in the future.
Feel free to ping me at any point 😉
Yes, it took a while also to me to figure that out. PyPI, in theory, could be used also to distribute non-python resources. If wheels are distributed properly (and I mean that C++ projects include the entire install tree, including headers + libraries + exported targets), by pointing Another project I found using a similar approach is CasADI. You can install it and inspect its folder inside site-packages. The case of manif and pybind11 is easier because they are header-only libs, the compiler does not play a big role. However, I want to point out again that projects distributing in this way their binaries must use the new PEP600 standard. Otherwise, manylinux1 and manylinux2014 will use a too old gcc version that does not have C++11 ABI and bad things can happen (robotology/idyntree#776). |
For future reference, I've found another explanation of the inclusion of headers / libs / targets etc inside the wheel in pybind's documentation. |
I'm very glad to see that Python support #187 has been introduced recently in #201 (previous #179).
I'm opening this issue to discuss / propose the possibility to support distributing the project through PyPI. For those that are not very familiar with the process, there are few incremental steps that could be done, each of them extending more and more the support. To begin with, I'd point interested readers to https://packaging.python.org/tutorials/packaging-projects/ and Python Packaging User Guide.
The main background to understand this issue is grasping the differences between a source distribution (aka
sdist
) and a binary distribution (aka wheel orbdist_wheel
).As first step, it would be nice distributing
manifpy
in the form of a source distribution. If the project would introduce the support ofpypa/setuptools_scm
, the source distribution would be no more no less than a compressed version of the git repository. When users executepip install manif
, it would download thesdist
and runsetup.py
. The process is quite straightforward.Of course, it is clear to see that in this case, and as documented in #201,
setup.py
expects to find some dependency in the system (eigen, cmake, pybind). Here come PEP517 and PEP518 to help us. In few words, during thepip install manif
process, they create an isolated Python environment (e.g. in/tmp
) and install the build dependencies before building thesdist
downloaded from PyPI. In this way, cmake and pybind can be automatically installed, and users do not have to manually install them in their system. AFAIK, eigen is not in PyPI, therefore for the moment it needs to be found in the system. Alternatively, a simpleFetchContent
logic could be added to the repo.A very useful tool to implement without much effort all what I explained above is
diegoferigo/cmake-build-extension
, that acts as a bridge between CMake-based projects and setup.py. Disclaimer: I'm the author of the package. Beyond the official documentation, a MWE that also supports PEP600 wheels and implements an automatic CI/CD pipeline isdiegoferigo/gazebo-yarp-synchronizer
.Right now
manif
only installs the shared library generated by pybind11 in the site-package folder.cmake-build-extension
enables creating a real Python package (with an__init__.py
) and also installing the exported CMake project (headers, libraries, cmake files), so that other packages could import and consume it. The same approach is implemented in thepybind11
package itself, beyond other ones.@artivis I liked to ask if you'd be interested in implementing something similar in manif. If needed, I can provide support and guidance.
Further information can be found in the downstream ami-iit/bipedal-locomotion-framework#296.
The text was updated successfully, but these errors were encountered: