Globs in extra paths

[itsdani]

I'm trying to setup a python monorepo build with bazel, and I'm struggling to make pylance recognize the library imports. I have managed to make it work by adding e.g. <bazel-myproject>/external/<my-pip-dependencies>/pypi__numpy to python.analysis.extraPaths, but adding a line for each dependency on every computer we use doesn't seem to be right.

My idea/request is that if we could add something like dependencies/*/ or dependencies/pypi__*/ to the extra paths, it would solve this problem entirely. The expected behavior would be to add every (matching) directory under dependencies/ to the extra paths.

[judej]

Thanks for the request @itsdani, we will review the request and look at what it would take to get this functionality

[jakebailey]

Copying my info here from microsoft/pyright#1684 (comment)

One consideration is that file watching logic for this becomes a little complicated because 1) we may need special handlers for parent directories of globbed items to know when the paths have to be regenerated, and 2) glob patterns may match files which we don't want in extraPaths (besides maybe zips if those ever get supported).

[dermidgen]

This is pretty critical for supporting monorepos with multiple python services.

Support for this feature would also resolve many of the open issues related to poetry with the use of local envs.

My teams currently address this challenge with a bash script that is run whenever a new python service or library is added.

#!/bin/bash
shopt -s dotglob
shopt -s nullglob

VENV_PATHS_ARR=(*/*/.venv/lib/*/site-packages/)
VENV_PATHS=$(printf ",\"%s\"" "${VENV_PATHS_ARR[@]}")
EXTRA_PATHS=${VENV_PATHS:1}

jq ".\"python.analysis.extraPaths\" = [${EXTRA_PATHS}]" .vscode/settings.json > .vscode/settings.tmp
mv .vscode/settings.tmp .vscode/settings.json 

[jakebailey]

Support for this feature would also resolve many of the open issues related to poetry with the use of local envs.

Can you clarify what you mean? I wouldn't expect anyone to use extraPaths to add paths that point to an interpreter.

[dermidgen]

We found extraPaths was the simplest way to support import resolution for small-medium monorepos. For larger monorepos or where granular ACLs are required, we found multi-root workspaces to be useful. However, even with multi-root workspaces, some roots may contain 10's to hundreds of services or libraries.

If there's a better approach than extraPaths - like actually switching envs - I would be very pleased to follow that. The downside to this approach is that imports will resolve regardless of which project your in.

With poetry and the following scaffolding:

./
...
./libs
./libs/lib-one
./libs/lib-one/poetry.toml
./libs/lib-one/pyproject.toml
./libs/lib-one/.venv
...
./stack
./stack/service-one
./stack/service-one/poetry.toml
./stack/service-one/pyproject.toml
./stack/service-one/.venv
...

Using poetry.toml

[virtualenvs]
in-project = true

[jakebailey]

You may be interested in https://github.com/microsoft/pyright/blob/main/docs/configuration.md#execution-environment-options, though I don't know to what extent it will have the behavior you're seeking. Execution environments are what @erictraut's team uses to manage their monorepo.

[dermidgen]

Execution environments are indeed helpful, in that it resolves the issue of resolving imports from outside the project path. This does require switching from .vscode/settings.json to pyproject.toml at the root of the repo - no biggie.

Unfortunately, since it doesn't allow setting venv for each execution environment 🤭, we still need to set extraPaths for each execution environment. Which, yes, is better than using .vscode/settings.json. However, since we need to set extraPaths we'll still need a helper script to do a glob drill down for "*/*/.venv/lib/*/site-packages/".

So I guess, if we can't get executionEnvironments to support an override for venv, we'd still greatly appreciate glob support for extraPaths.

[jakebailey]

The execution environment model is not suited to have separate interpreters, unfortunately. There used to be support, but it was broken as it was never intended to work. Multi-root runs separate analyzers, so they are able to have separate interpreters. Execution environments are within an analyzer, and can only have one.

We need to come up with a spec to better reconcile the idea of execution environments and multiroot into something that can capture how people are using. I think details on how your monorepo is laid out and what you're expecting would be helpful for that (though, this issue was about globbing, not that redesign 🙂)

[dermidgen]

@jakebailey I'd be happy to support that effort in any way; we can move the interpreters/monorepo discussion. Here's an example of what I'm currently testing wtih. https://github.com/dermidgen/python-monorepo

I yield the thread back to globbing 😁

[danmx]

any updates?

[judej]

Moving this issue to discussion as an enhancement request for comments and upvotes.

[oselcuk-iqm]

Are there any updates on this issue? It's quite painful to use VSCode with a bazel python repo without this feature

[gdippolito]

I ended up creating a script to manipulate .vscode/settings.json for reference https://gist.github.com/gdippolito/72fc9e099d9c0d99ff0d79fac37864f9

We run the script after a bazel rule that will pull down all the dependencies.

[thomassajot]

It would be great to have this feature.

[guntribam]

Just keep the begging alive

[sulekz]

Additional begging.

[erictraut]

For those of you who are interested in this feature, it would be useful if you could provide some details about the problem you're trying to solve. That would allow us to consider other potential solutions.

There's a good reason why we don't support globs in extraPaths today. The order of extraPaths is significant. This setting defines the specific order in which imports are resolved. This ordering matters, for the same reason that the order of sys.path matters at runtime. Globs don't allow for any ordering. They are resolved in whatever order the file system happens to return directories or files. For that reason, it's not a good idea to use globs for any configuration settings that are order-dependent.

[misha-antonenko]

The glob-specified paths could be sorted, and the ordering would be neither too surprising nor inconsistent. It anyway sounds like a bad a idea to rely on globs when you want definite ordering

[itsdani]

My original problem was that when I set up a monorepo project with Bazel, it collects the dependencies of the projects into directories like ./dependencies/pip/<package-name>/. So the executables are not in a single directory, and the list of directories can easily grow/change if someone adds a new dependency. In this scenario I would not care at all about the order of the extraPaths entries, because they are all different packages that should not conflict.

I suspect that extraPaths was never intended to do this kind of dependency inclusion. Virtual envs would be a better thing to do, but (when I tried doing this) Bazel did its own thing without using venvs.

[erictraut]

If bazel places all dependences into a directory named ./dependencies/pip/<package-name>, can't you simply add dependencies/pip to your extraPaths?

Are you able to run your code in the environment that bazel creates? In other words, do imports resolve at runtime within this environment? If so, do you know how that works? For example, is bazel setting the PYTHONPATH environment variable to this directory (or directories) so the Python interpreter updates sys.path at startup?

[triplepoint]

For what it's worth, I have exactly the same use case: Bazel has an opinionated way of managing Python dependencies, and the only current solution is to regularly regenerate extraPaths with a shell script. Globbing would be the most obvious way to dynamically declare this list, but I'd be curious to hear alternative suggestions. Oh, and I'm one of the poor souls stuck using bazel in a polyrepo (not monorepo) scenario, so I think it's best to think of the bazel/globbing and the monorepo discussions as separate but related.

…

On Wed, Aug 23, 2023, 01:04 Daniel Segesdi ***@***.***> wrote: My original problem was that when I set up a monorepo project with Bazel, it collects the dependencies of the projects into directories like ./dependencies/pip/<package-name>/. So the executables are not in a single directory, and the list of directories can easily grow/change if someone adds a new dependency. In this scenario I would not care at all about the order of the extraPaths entries, because they are all different packages that should not conflict. I suspect that extraPaths was never intended to do this kind of dependency inclusion. Virtual envs would be a better thing to do, but (when I tried doing this) Bazel did its own thing without using venvs. — Reply to this email directly, view it on GitHub <#2712 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACD4G7EU5GXSIHFVWFZEA3XWW2P5ANCNFSM5WYQKR4A> . You are receiving this because you are subscribed to this thread.Message ID: <microsoft/pylance-release/repo-discussions/2712/comments/6799676@ github.com>

[rugged-robotics]

One use case where this is useful is for people using ROS1 devel workspaces. ROS1 uses a model of masking workspaces for installation and development, with the primary installation workspace manually installing files to a known location, and the development workspaces linked to those shadowing the installed files via some path magic and link to the package directories directly.

Since the development workspace contains multiple packages all at the same level (all code is in workspace/src/ directory in discrete packages) as well as each ros package having it's own src directory which contains the python packages it defines; there's no real concern over resolution order for this application, but there is a need to effectively product a glob of workspace/src/*/src/ or workspace/src/**/src/ to have an extra path pointing inside each packages src directory.

It would be useful for this application is the glob in extra paths expanded in place in the list of extra paths, so we could prioritize python packages in the system and installation spaces, but all packages within the glob have arbitrary dependency order.

This may all seem somewhat convoluted, but it's the framework this system provides

[judej]

This has been added to our backlog. You can track progress at #5364

[adanielnoel]

Still an issue. Managed to get a workaround similar to gdippolito's

I ended up creating a script to manipulate .vscode/settings.json for reference https://gist.github.com/gdippolito/72fc9e099d9c0d99ff0d79fac37864f9

We run the script after a bazel rule that will pull down all the dependencies.

Instead, I created a py_binary rule that must depend on all external python packages I use like so:

py_binary(
    name = "vscode_import_generator",
    srcs = ["vscode_import_generator.py"],
    deps = [
        "@pypi//numpy",
        ...
    ],
)

A bit annoying to have to keep it updated but couldn't find a better way since we can't glob dependencies in Bazel.

This is my modified version of the script: https://gist.github.com/gdippolito/72fc9e099d9c0d99ff0d79fac37864f9

[heejaechang]

This discussion seems to be divided into two requests: one for Mono users and the other for Bazel users. We probably need two separate solutions for each group. For Mono repo users, this will probably solve your issue?