How to use pyproject.toml with src-layout and __main__.py?

[dragondive]

In the flat layout, I can create and execute a local module with (python -m mypkg) by including a __main__.py inside mypkg:

project_root_directory
├── pyproject.toml
├── ...
└── mypkg/
    ├── __main__.py
    └── ...

Is it possible to achieve the same using the src-layout?

If I create the src-layout like this:

project_root_directory
├── pyproject.toml
├── ...
└── src/
    ├── mypkg/
    └── __main__.py

Then the module name becomes src and can be executed as python -m src, which is obviously not what I want. I could move the __main__.py inside mypkg:

project_root_directory
├── pyproject.toml
├── ...
└── src/
    └── mypkg/
            ├── __main__.py
            └── subpkg/
                    └── stuff.py

and in pyproject.toml:

[tool.setuptools.packages.find]
where = [
    "src/mypkg"
]

This "works", but it also creates (what looks to me as) another problem. It installs the following modules:

    c:\...\python\python311\lib\site-packages\build\lib\mypkg\__main__.py
    c:\...\python\python311\lib\site-packages\build\lib\mypkg\subpkg\stuff.py
    c:\...\python\python311\lib\site-packages\mypkg-0.0.1.dist-info\*
    c:\...\python\python311\lib\site-packages\mypkg\__main__.py
    c:\...\python\python311\lib\site-packages\mypkg\subpkg\stuff.py
    c:\...\python\python311\lib\site-packages\src\mypkg\__main__.py
    c:\...\python\python311\lib\site-packages\src\mypkg\subpkg\stuff.py
    c:\...\python\python311\lib\site-packages\subpkg\stuff.py

Notice there's some duplication(?) of installed modules, once under mypkg and once under src\mypkg.

Version information

OS: Windows 11
python: 3.11.1
pip: 22.3.1

Commands used

• To create the virtual environment: python -m venv venv_mypkg
• To build and install the package: python -m pip install .

[abravalheri]

Hi @dragondive, please see my 3 comments below:

1. I believe that in your case the correct configuration is where = ["src"], according to the docs in https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#finding-namespace-packages.
   Setuptools will look for packages (i.e. directories) inside where, but this does not include the value of where itself, or any module (i.e. Python files) 1 level inside of it.
   Note however that you don't need to add this configuration. Setuptools will automatically look for packages inside a src folder if you simply omit all the configuration for tool.setuptools.packages as you can see in the following example:

   > docker run --rm -it python:3.11-windowsservercore powershell
   
   cd $env:TEMP
   mkdir project_root_directory
   cd project_root_directory
   mkdir src
   mkdir src/mypkg
   mkdir src/mypkg/subpkg
   new-item src/mypkg/__main__.py
   new-item src/mypkg/subpkg/stuff.py
   
   new-item pyproject.toml
   @"
   [build-system]
   build-backend = "setuptools.build_meta"
   requires = ["setuptools >= 67.6"]
   
   [project]
   name = "mypkg"
   version = "0.0.1"
   "@ | add-content -Path pyproject.toml
   
   python -m venv .venv_mypkg
   .venv_mypkg/Scripts/python -m pip install -U 'pip==22.3.1'
   .venv_mypkg/Scripts/python -m pip install .
   
   ls .\.venv_mypkg\Lib\site-packages
   #
   #     Directory: C:\Users\ContainerAdministrator\AppData\Local\Temp\project_root_directory\.venv_mypkg\Lib\site-packages
   #
   # Mode                LastWriteTime         Length Name
   # ----                -------------         ------ ----
   # d-----        3/24/2023  10:11 AM                mypkg
   # d-----        3/24/2023  10:11 AM                mypkg-0.0.1.dist-info
   # d-----        3/24/2023  10:11 AM                pip
   # d-----        3/24/2023  10:11 AM                pip-22.3.1.dist-info
   # d-----        3/24/2023  10:11 AM                pkg_resources
   # d-----        3/24/2023  10:11 AM                setuptools
   # d-----        3/24/2023  10:11 AM                setuptools-65.5.0.dist-info
   # d-----        3/24/2023  10:11 AM                _distutils_hack
   # -a----        3/24/2023  10:11 AM            151 distutils-precedence.pth
   
   tree .\.venv_mypkg\Lib\site-packages\mypkg
   # ...
   # <...>\.VENV_MYPKG\LIB\SITE-PACKAGES\MYPKG
   # │   __main__.py
   # │
   # ├───subpkg
   # │   │   stuff.py
   # │   │
   # │   └───__pycache__
   # │           stuff.cpython-311.pyc
   # │
   # └───__pycache__
   #         __main__.cpython-311.pyc

2. In the example you gave, the configuration should result in only subpkg being added to site-packages (+ a mandatory .dist-info folder), due to the reasons I explained in (1.). When I tried to reproduce your example, that is exactly the behaviour I observe, with no duplication (please see below). So probably there are more factors involved in the behaviour you are seeing... Maybe files from old installations don't get "garbage collected" by pip when it is re-installing the project? Maybe the project name accidentally changed, so pip thought it did not need to remove old files?

   > docker run --rm -it python:3.11-windowsservercore powershell
   
   cd $env:TEMP
   mkdir project_root_directory
   cd project_root_directory
   mkdir src
   mkdir src/mypkg
   mkdir src/mypkg/subpkg
   new-item src/mypkg/__main__.py
   new-item src/mypkg/subpkg/stuff.py
   
   new-item pyproject.toml
   @"
   [build-system]
   build-backend = "setuptools.build_meta"
   requires = ["setuptools >= 67.6"]
   
   [project]
   name = "mypkg"
   version = "0.0.1"
   
   [tool.setuptools.packages.find]
   where = ["src/mypkg"]
   "@ | add-content -Path pyproject.toml
   
   python -m venv .venv_mypkg
   .venv_mypkg/Scripts/python -m pip install -U 'pip==22.3.1'
   .venv_mypkg/Scripts/python -m pip install .
   
   ls .\.venv_mypkg\Lib\site-packages
   # 
   # 
   #     Directory: C:\Users\ContainerAdministrator\AppData\Local\Temp\project_root_directory\.venv_mypkg\Lib\site-packages
   # 
   # 
   # Mode                LastWriteTime         Length Name
   # ----                -------------         ------ ----
   # d-----        3/24/2023   9:57 AM                mypkg-0.0.1.dist-info            # <--- as expected
   # d-----        3/24/2023   9:56 AM                pip
   # d-----        3/24/2023   9:56 AM                pip-22.3.1.dist-info
   # d-----        3/24/2023   9:56 AM                pkg_resources
   # d-----        3/24/2023   9:56 AM                setuptools
   # d-----        3/24/2023   9:56 AM                setuptools-65.5.0.dist-info
   # d-----        3/24/2023   9:57 AM                subpkg                           # <--- as expected
   # d-----        3/24/2023   9:56 AM                _distutils_hack
   # -a----        3/24/2023   9:56 AM            151 distutils-precedence.pth

[dragondive]

Hi @abravalheri, thank you very much for those valuable hints. In the meantime, I also studied the pip package manager as well as setuptools in some more depth.

You are correct, it was indeed a case of "leftovers" from previous installs that were not properly cleaned up. I had been experimenting with various directory layouts to self-learn the pyproject.toml based configuration approach. This led to pip retaining some stuff from those other layouts. After understanding the problem, I created a fresh venv, and then I could get it working properly.