Symlink install of data_files for python setuptools packages

[stonier]

This is especially useful for, e.g. ros2 launch files that need to reside in share/<package_name>. Currently, if you have

    ....
    data_files=[
        ('share/' + package_name, ['package.xml', 'my.launch.py']),
    ],
    ....

Then package.xml gets symlinked but the launch files do not:

package.bash
package.ps1
package.sh
my.launch.py
package.xml -> /my_workspace/src/my_package/package.xml
package.zsh

Symlink install handling is in colcon_core/task/python/__init__.py. @dirk-thomas mentioned that pulling the information from setup.py to unwind the install and replace with symlinks is difficult and potentially unreliable (depending on whatever other programmatic mischief is going on in setup.py) and a better place to define the information is in setup.cfg.

options.data_files is not supported in setuptools on bionic though (v39.0.1). Seems the version necessary may be v40.5.0 (setuptools issue 1522) and may have issues with dashes.

[rotu]

is this related to, e.g. ament/ament_lint#151 or is it just handling of data_files?

[rotu]

I had a problem with getting my package keystroke to run when installed with --symlink-install that I tracked down to a bug in colcon fixed in PR 66. Does your package have a setup.cfg or just a setup.py?

[rotu]

Okay, I understand now. Python's setup tools generally install the files specified in 'data_files' via distutils/command/install_data.py, which straight-up copies the files. To do this cleanly with setup.py would either involve extending setuptools and/or replacing the call to setup.py install_data with some other tool.

[dirk-thomas]

Since newer versions of setuptools (available in Debian 9 Stretch as well as Ubuntu 19.04 Disco) support this natively we won't implement any custom logic for older platforms. Therefore I am going ahead and close this ticket.

[stonier]

Do you happen to have a link to some more information @dirk-thomas? I'm not having much luck discovering anything on google.

[stonier]

Just for reference for others, I've been working around this problem with launch files by putting implementations of the launch methods in the package modules (which indirectly get symlink installed) and simply making a one line call out to it from the launch script. Since the one line call out rarely changes, it just needs the one install.

[emersonknapp]

I think this issue should not have been closed, it has not been resolved. Maybe @dirk-thomas misread the original issue as basic data_files install support. @cottsay (assuming you're the de facto maintainer now?) Perhaps we could reopen it so as not to mislead?

[cottsay]

We can reopen, but nothing has really changed here. The underlying build system (setuptools) still does not support this (here's the spot).

I'm not able to find any evidence that there is a supported mechanism to do this even in newer versions of setuptools, which was referenced as the reason it was closed to begin with.

[emersonknapp]

Hmm, is there any way we could clearly note the limitation (like, notice --symlink-install and data_files then print a warning)? I'll admit that even as an experienced ROS 2 dev I was confused by this the other day when my launchfile behavior didn't change after being edited.

I think the part that's most confusing to a user is the disparity between CMake and setuptools. Since in ROS 1 we only had catkin cmake, so there wasn't the possibility for a difference. It is inevitable that, by supporting totally different build tools via colcon, we invite differences in behavior between them. But, symlinked launchfiles feel like one of those really basic development loops that will have been ingrained into any longtime ROS user. That's my argument for keeping this open as a bug, even if I don't have an answer for how to solve it.

[cottsay]

I've opened a draft PR (#592) which might be a possible solution here. It digs into setuptools a little more than I'd like it to, but it seems to function how we want it to.

Please take a look and provide feedback if it works for you.