package_dir vs where

[codeandfire]

Hi @abravalheri,

I was just going through the Package Discovery docs page again, and I noticed that the example codes in the Custom Discovery section for setup.cfg and setup.py specify both the package_dir and where arguments, whereas this is not the case with the codes for pyproject.toml: they specify only the where argument, not package_dir.

What I mean is: consider the simple package discovery section -

setup.cfg

[options]
packages = find:
package_dir =
    =src

[options.packages.find]
where = src
include = pkg*
exclude = additional

setup.py

setup(
    # ...
    packages=find_packages(
        where='src',
        include=['pkg*'],
        exclude=['additional'],
    ),
    package_dir={"": "src"}
    # ...
)

pyproject.toml

[tool.setuptools.packages.find]
where = ["src"]
include = ["pkg*"]
exclude = ["additional"]
namespaces = false

Basically the pyproject.toml snippet does not specify the package_dir argument but the other two do, along with the where argument.
The snippets for the namespace package discovery section are similar:

setup.cfg

[options]
package_dir =
    =src
packages = find_namespace:

[options.packages.find]
where = src

setup.py

setup(
    # ...
    packages=find_namespace_packages(where='src'),
    package_dir={"": "src"}
    # ...
)

pyproject.toml

[tool.setuptools.packages.find]
where = ["src"]

Again the same situation holds.

Therefore I think there is some inconsistency here and should be corrected ...
Noting that both package_dir and where convey the same information (the src directory), perhaps one of them is redundant?
I wanted to know whether both arguments are required, and if not, which argument is more preferred than the other?

[codeandfire]

I sort of missed noticing this while drafting the original PR for the Package Discovery page.
Post your response, I can draft a new PR to make the necessary changes and submit it.

[abravalheri]

Hi @codeandfire, this is a new development... When I was implementing support for configuration via pyproject.toml someone pointed out that package_dir is difficult (a very overkill config that most of the people underuse just for the src-layout scenario) and suggested to automatically take into consideration what is passed via where.

I definitely implemented that for pyproject.toml, I don't remember if I did the same for setup.cfg, but I don't think it is possible to do that for setup.py.

[abravalheri]

Hi @codeandfire, what happens is that the package_dir will be backfilled in the case if pyproject.toml (maybe setup.cfg) but not setup.py.

When using the setup.py script the find_packages() or find_namespace_packages() functions are evaluated eagerly on the spot. By the time the setup() function is called, the information about the original value for the where argument is no longer available and the package_dir will not be backfilled.

[codeandfire]

Oh, okay, I get it now. Thank you for the clarification!

[codeandfire]

Hi @abravalheri, I know I am reviving what is sort of an old discussion... But I noticed today that while setup.py definitely needs package_dir along with where, setup.cfg seems to be able to do without package_dir.

Here is a test:

% mkdir project_root_directory
% cat > project_root_directory/pyproject.toml <<EOF
heredoc> [build-system]
heredoc> requires = ["setuptools>=60.0", "wheel"]
heredoc> build-backend = "setuptools.build_meta"
heredoc> EOF

% mkdir project_root_directory/timmins
% rmdir project_root_directory/timmins 
% mkdir -p project_root_directory/src/timmins
% cat > project_root_directory/src/timmins/__init__.py <<EOF
heredoc> def hello_world():
heredoc>     print('Hello world')
heredoc> EOF

% tree project_root_directory 
project_root_directory
├── pyproject.toml
└── src
    └── timmins
        └── __init__.py

2 directories, 2 files

% cat > project_root_directory/setup.cfg <<EOF
heredoc> [metadata]
heredoc> name = timmins
heredoc> version = 0.1.0
heredoc> 
heredoc> [options]
heredoc> packages = find:
heredoc> 
heredoc> [options.packages.find]
heredoc> where = src
heredoc> EOF

% cd project_root_directory && pip3 install .
Processing /home/codeandfire/tmp/package-dir/project_root_directory
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: timmins
  Building wheel for timmins (pyproject.toml) ... done
  Created wheel for timmins: filename=timmins-0.1.0-py3-none-any.whl size=1132 sha256=653c0a5d1c26f9bae304c81807e61fddb99f4f5aa666b38680f6f97f8cf3b084
  Stored in directory: /tmp/pip-ephem-wheel-cache-af3r8mtn/wheels/ef/df/2f/0d648436df5b9c093f97a2715c29beb8748f1abb1a39a34eb1
Successfully built timmins
Installing collected packages: timmins
Successfully installed timmins-0.1.0

% cd ~
% python3
Python 3.10.5 (main, Jun  6 2022, 18:49:26) [GCC 12.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import timmins
>>> timmins.hello_world()
Hello world
>>> exit()

And for namespace packages:

% tree
.
├── pyproject.toml
├── setup.cfg
└── src
    └── timmins
        └── __init__.py

2 directories, 3 files

% mkdir src/timmins/foo
% mv src/timmins/__init__.py src/timmins/foo 

% cat setup.cfg 
[metadata]
name = timmins
version = 0.1.0

[options]
packages = find:

[options.packages.find]
where = src

% sed -i 's/packages = find:/packages = find_namespace:/' setup.cfg 
% cat setup.cfg 
[metadata]
name = timmins
version = 0.1.0

[options]
packages = find_namespace:

[options.packages.find]
where = src

% tree
.
├── pyproject.toml
├── setup.cfg
└── src
    └── timmins
        └── foo
            └── __init__.py

3 directories, 3 files

% pip3 install .
Processing /home/codeandfire/tmp/package-dir/project_root_directory
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: timmins
  Building wheel for timmins (pyproject.toml) ... done
  Created wheel for timmins: filename=timmins-0.1.0-py3-none-any.whl size=1144 sha256=68026ea4f3bcb48a2a561a9fbe083d903139c6106f340c6099ff7dd65a3180d0
  Stored in directory: /tmp/pip-ephem-wheel-cache-75kmza9l/wheels/ef/df/2f/0d648436df5b9c093f97a2715c29beb8748f1abb1a39a34eb1
Successfully built timmins
Installing collected packages: timmins
  Attempting uninstall: timmins
    Found existing installation: timmins 0.1.0
    Uninstalling timmins-0.1.0:
      Successfully uninstalled timmins-0.1.0
Successfully installed timmins-0.1.0

% cd ~
% python3
Python 3.10.5 (main, Jun  6 2022, 18:49:26) [GCC 12.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import timmins.foo
>>> timmins.foo.hello_world()
Hello world
>>> exit()

[abravalheri]

Thank you very much for investigating this deeply @codeandfire.

So would it be the case that we have the following scenarios?

1. Barebones setup.py/setup.cfg/pyproject.toml (no packages or package_dir configuration).
   • Relies on auto-discovery. src and flat-layouts are automatically detected.
2. setup.cfg with packages = find:/find_namespace: or pyproject.toml with packages = {find = ...}
   • package_dir will be backfilled with the value of where
3. setup.py with packages = [...] or packages = find_packages()/find_namespace_packages()
   • The value for packages is resolved eagerly and the value for package_dir needs to be explicitly specified by the user.

[codeandfire]

Yes, I believe you are correct on all three points.