Windows with --symlink-install picking up package resource folders as Python packages.

[danzimmerman]

Expected Behavior

The resource marker files in ament_python packages do not interfere with other Python tools.

Actual Behavior

On Windows, if I use colcon build --merge-install --symlink-install to build a workspace with ament_python packages and source the workspace's setup.bat, the package's resource folder is picked up as an importable Python package.

This causes errors in other packages.

(humble) c:\Code\dbgdump\test_ws>python -c "import resource; print(resource.__spec__)"
ModuleSpec(name='resource', loader=<_frozen_importlib_external._NamespaceLoader object at 0x0000020E1B72EC10>, submodule_search_locations=_NamespacePath(['c:\\Code\\dbgdump\\test_ws\\build\\test_pkg\\resource']))

Note that the imported module is in the workspace's build folder:

c:\\Code\\dbgdump\\test_ws\\build\\test_pkg\\resource

The resource package is supposed to be a Unix-specific Python package that I guess is in the standard library.

Several other Python packages test only for an ImportError for the resource package as a test for non-Unix systems. For example:

• https://github.com/jupyter/notebook/blob/v6.4.11/notebook/notebookapp.py#L35
• https://github.com/prometheus/client_python/blob/master/prometheus_client/process_collector.py#L7

However, if there is an importable resource package that isn't https://docs.python.org/3/library/resource.html, it raises an AttributeError instead.

I first encountered this issue because Jupyter notebooks seem to use the resource module, which means I can't properly install and use Jupyter notebooks in a fully-sourced ROS2 environment on Windows ROS2.

See jupyter/notebook#5817 (comment) for earlier details.

Detailed Reproduction

I did a step-by-step reproduction in an otherwise empty workspace below.

The problematic outcome is annotated with ❌... the errors annotated with ✅ represent expected behavior on Windows.

It's specifically the use of --symlink-install that triggers the behavior.

(humble) c:\Code\dbgdump\test_ws>cd src
(humble) c:\Code\dbgdump\test_ws\src>ros2 pkg create test_pkg --build-type ament_python
(humble) c:\Code\dbgdump\test_ws\src>tree /f
Folder PATH listing for volume Windows-SSD
Volume serial number is BC74-9D52
C:.
└───test_pkg
    │   package.xml
    │   setup.cfg
    │   setup.py
    │
    ├───resource
    │       test_pkg
    │
    ├───test
    │       test_copyright.py
    │       test_flake8.py
    │       test_pep257.py
    │
    └───test_pkg
            __init__.py
            
(humble) c:\Code\dbgdump\test_ws\src>cd ..
(humble) c:\Code\dbgdump\test_ws>colcon build --merge-install --symlink-install

# Before we source the workspace, we get a ModuleNotFoundError, which is fine
# This is considered a sign that we're on Windows for other packages looking for the resource module
# like https://github.com/prometheus/client_python
(humble) c:\Code\dbgdump\test_ws>python -c "import resource"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'resource' ✅ 

# however, when we source the workspace, we have an issue 

(humble) c:\Code\dbgdump\test_ws>install\setup.bat
(humble) c:\Code\dbgdump\test_ws>python -c "import resource"

# nothing happens here, because it exists! What is it? The resource folder from test_pkg
(humble) c:\Code\dbgdump\test_ws>python -c "import resource; print(resource.__path__)"
_NamespacePath(['c:\\Code\\dbgdump\\test_ws\\build\\test_pkg\\resource']) 

# Packages that test for ImportError as a platform selector will now fail.
# We get an AttributeError if we use methods that exist in the Unix resource module
(humble) c:\Code\dbgdump\test_ws>python -c "import resource; print(resource.getpagesize())"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: module 'resource' has no attribute 'getpagesize' ❌ 

# We're fine if we just use colcon build
(humble) c:\Code\dbgdump\test_ws>rd /s /q build install log
(humble) c:\Code\dbgdump\test_ws>colcon build 
(humble) c:\Code\dbgdump\test_ws>install\setup.bat
(humble) c:\Code\dbgdump\test_ws>python -c "import resource; print(resource.getpagesize())"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'resource' ✅ 

# It seems like colcon build --merge-install is fine as well
(humble) c:\Code\dbgdump\test_ws>rd /s /q build install log
(humble) c:\Code\dbgdump\test_ws>colcon build --merge-install
(humble) c:\Code\dbgdump\test_ws>install\setup.bat
(humble) c:\Code\dbgdump\test_ws>python -c "import resource; print(resource.getpagesize())"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'resource' ✅ 

Machine Info

I have ROS2 installed via Robostack

Edition	Windows 11 Home
Version	22H2
Installed on	‎5/‎12/‎2022
OS build	22622.601

Python 3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:50:36) [MSC v.1929 64 bit (AMD64)] on win32

(humble) c:\Code\dbgdump\test_ws>mamba list | findstr ament-index
ros-humble-ament-index-cpp 1.4.0            py39he8739fe_1    robostack-humble
ros-humble-ament-index-python 1.4.0            py39h4c0ac80_1    robostack-humble

I'm considering some upstream PRs to fix the tests in the affected packages too, but since resource is a standard library package, it seems pretty valid to presume that no one would shadow its name in Windows unless their intent was to provide the same API for a special case.

I have not reproduced this on all my Windows machines, so I'll report back if I find that this is something peculiar about this machine and ROS2 installation.

Any advice for workarounds is appreciated. For now I'm just going to locally hack the offending Python libraries to test for AttributeError as well.