Add distutils package needed by colcon.

[marcoag]

Dev jobs are failing like: https://build.ros2.org/view/Rdev/job/Rdev__rqt_srv__ubuntu_noble_amd64/2/console with:

ERROR:colcon.colcon_core.extension_point:Exception loading extension 'colcon_core.package_identification.python_setup_py': No module named 'distutils'

This PR adds python3-distutils to the devel_task.Dockerfile.em file in an effort to fix the missing dependency.

[cottsay]

I'm willing to bet that this is a regression caused by #1026. The distutils module is normally provided by setuptools.

[cottsay]

Also, somewhat ironically, this doesn't appear to fix the problem. In a fresh noble VM, the python3-distutils package appears to install the module for Python 3.11, not 3.12 as is required.

[cottsay]

Oh, this is fun, I think I figured it out.

1. We're installing the "upstream" colcon packages provided by Ubuntu instead of the ones we provide in the ROS 2 repositories. We've discussed how to make apt prefer our colcon packages over the upstream ones, but that work still hasn't landed (FYI @nuclearsandwich @j-rivero)
2. The upstream python3-colcon-python-setup-py has a bug and is missing a runtime dependency on python3-setuptools. I'd suppose we should file a ticket.
3. The pip install of setuptools which was removed in Adding break-system-packages pip option for noble in devel_task #1026 was previously satisfying this requirement.

All three factors were needed for this behavior to manifest.

My recommendation is to install python3-setuptools here instead of python3-distutils. There are other errors in the job referring to setuptools, so even if python3-distutils wasn't bugged on Noble, we'd still need setuptools installed. After we resolve (1), we can revert this change.

[nuclearsandwich]

My recommendation is to install python3-setuptools here instead of python3-distutils. There are other errors in the job referring to setuptools, so even if python3-distutils wasn't bugged on Noble, we'd still need setuptools installed. After we resolve (1), we can revert this change.

I think that in the end we will likely need both the apt setuptools and distutils packages since Debian splits the modules into separate even though they're both part of setuptools upstream.

[marcoag]

python3-setuptools seems to have a dependency on python3-distutils though:

$ apt-cache depends python3-setuptools
python3-setuptools
  Depends: python3-pkg-resources
  Depends: python3-distutils
  Depends: <python3:any>
    python3:i386
    python3
  Suggests: python-setuptools-doc

[nuclearsandwich]

I've put up #1028 which restores @marcoag's original proposal while we figure out the rest of what should happen in order to remove this pip installation of setuptools.

[cottsay]

python3-setuptools seems to have a dependency on python3-distutils though:

It's inconsequential. I have no idea why they did that.

Installing python3-distutils alone doesn't work for Python 3.12:

$ sudo apt install -y python3-distutils
...
$ python3 -c "import distutils; print(distutils.__file__)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'distutils'

It does, however, work for Python 3.11:

$ python3.11 -c "import distutils; print(distutils.__file__)"
<string>:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
/usr/lib/python3.11/distutils/__init__.py

The python3-setuptools package bundles distutils and uses a .pth file to inject it into the Python module search:

$ dpkg -L python3-setuptools
...
/usr/lib/python3/dist-packages
/usr/lib/python3/dist-packages/_distutils_hack
/usr/lib/python3/dist-packages/_distutils_hack/__init__.py
/usr/lib/python3/dist-packages/_distutils_hack/override.py
/usr/lib/python3/dist-packages/distutils-precedence.pth
...
/usr/lib/python3/dist-packages/setuptools/_distutils
/usr/lib/python3/dist-packages/setuptools/_distutils/__init__.py
...
$ python3 -c "import distutils; print(distutils.__file__)"
/usr/lib/python3/dist-packages/setuptools/_distutils/__init__.py

[cottsay]

tl;dr - Ignore Ubuntu's python3-distutils package entirely, and assume that python3-setuptools provides the mechanism to import distutils. I do not believe that installing setuptools from pip is necessary, @nuclearsandwich.

[nuclearsandwich]

In a maintenance meeting, @cottsay and I have discussed the various fixes here: unconditionally installing setuptools, installing setuptools alongside the colcon package with the missing dependency, and since this is all arising because of the colcon version in upstream noble missing a runtime dependency on setuptools, updating our version of colcon-core so it has higher precedence than the upstream one again: That's being handled in ros-infrastructure/reprepro-updater#203 which is blocked on a deployment issue but moving forward. Let's hold off on either this PR or #1028 until that lands to see if it fixes this issue.

[cottsay]

see if it fixes this issue

Alright, I can now see python3-setuptools getting installed with colcon, but we finally hit the wall on our conflicts with the upstream colcon packages and we need to address that to move forward.

At present, our python3-colcon-core provides /usr/bin/colcon, which is provided by a package simply called colcon in upstream Debian. Without any metadata which indicates that they conflict, both packages are being installed and the installation fails during unpacking. Even if we have a pinning solution to make apt prefer our packages over upstream's, we should not allow apt to even try to install them both if we know the operation will fail.

I'm proposing that our python3-colcon-core "Replaces" the colcon package from upstream: colcon/colcon-core#628
This should allow the installation of upstream's colcon package to proceed, but the conflicting files will come from our python3-colcon-core automatically.

[cottsay]

The aforementioned "Replaces" change to colcon-core has resolved the issue, and Rdev/Rpr jobs appear to be turning over.