Configurable workspace indexing limit?

[lalten]

VS Code tells me [Warn - 1:02:37 PM] Workspace indexing has hit its upper limit: 2000 files

So I guess I'm hitting https://github.com/microsoft/pyright/blob/2477c9030f42d30ade1562f0a01ff20e5641df5c/packages/pyright-internal/src/analyzer/program.ts#L475

Can this limit be made a configurable pyrightconfig.json option?

This is in a workspace with a couple thousand Python files and the config json is checked in so excluding some dirs is not an option.
An alternative solution would be to have VS Code provide an exclude setting to pass through to Pylance/pyright.

[erictraut]

Workspace indexing is enabled only in pylance, so I'm going to transfer this issue to the pylance repo.

[jakebailey]

For the latter idea, see: #1150

We'd appreciate your vote on that if that interests you.

[ngutman]

Is the hard limit in-place to protect against memory issues? We have a huge mono-repo (~+4000 python files in the workspace) and the lack of indexing is very noticeable.

[misha-antonenko]

I can confirm this is a large issue with mono-repositories.

[thomassajot]

Is there a workaround for this issue?

[okke-formsma]

In some cases you might be able to exclude files that you don't need.

For example, I'm excluding our Django migrations files in pyrightconfig.json:

{   
    // There's a limit of 2000 files to index, so exclude migrations as they're never imported from other places.
    "exclude": [
      "**/migrations",
    ],
}

Would be nice to configure this in .vscode/pylance.json, but that's not possible yet: #2581

[jarshwah]

This issue has been around for a long while so I figured I'd give it another bump with some data from my system. Go To Symbol in Workspace doesn't work at all in my project. The logs indicate we have approx 35,000 source files which is approximately correct.

[Info  - 2:08:12 pm] (11868) Found 35542 source files
[Info  - 2:08:12 pm] (11868) Indexer background runner(4) root directory: /Users/josh/.vscode/extensions/ms-python.vscode-pylance-2023.1.21/dist (index)
[Info  - 2:08:12 pm] (11868) Indexing(4) started
[Info  - 2:08:13 pm] (11868) scanned(4) 356 files over 1 exec env
[Info  - 2:08:15 pm] (11868) indexed(4) 189 files over 1 exec env
[Info  - 2:08:15 pm] (11868) Indexing finished(4).
[Warn  - 2:08:17 pm] (11868) Workspace indexing has hit its upper limit: 2000 files

[ZFYL]

So this issue seems like a hard wall for moving to VSCode. Is there a way to build VSCode with a higher limit then 2000?

[erictraut]

I'm not sure what you mean by "build VSCode". Pyright is a type checker. Pylance is a language server that builds on pyright's functionality to provide additional functionality. This additional functionality includes background symbol indexing. Are you using pylance with VS Code? If so, pylance feature requests (like dynamic configurability of the indexing limit) can be posted to the pylance-release project.

[ZFYL]

actually it's all in Pyright, so only that needs an additional setup to make the 2000 limit configurable

[ZFYL]

It's hard coded into the code to avoid out of memory issues

Maybe we could build the whole package with a higher limit

[ZFYL]

this is the path

[smackesey]

Throwing my hat in the ring, we also have a large monorepo with over 2000 files, and this limit seems to cause inconsistent behavior with symbol renames and autocomplete. Please make it configurable.

[luabud]

This is now available through the "python.analysis.userFileIndexingLimit" setting. You can learn more about this setting in our docs: https://code.visualstudio.com/docs/python/settings-reference#_python-language-server-settings

[MarkValadez]

I seem to have a similar issue, where pylance is indexing 6k files. It is running out of memoryfrom reading the virtual environment, so I recon part of the environment goes without indexing. Is this normal behavior? I tried signalling the venv folder and excluding it from linting, but then i get missing import errors on standard python packages. Here is my pyproject.toml

[tool.mypy]
mypy_path = ".stubs"
strict = true
pretty = true
disallow_untyped_defs = true
disable_error_code = ["import-untyped"]
plugins = ["strawberry.ext.mypy_plugin"]


[tool.ruff]
src = [".", "src", "tests", "visory"]
extend-exclude = [".github", "**/__pycache__", "**/data"]
output-format = "full"
line-length = 90
indent-width = 4

[tool.ruff.format]
line-ending = "auto"
quote-style = "single"
indent-style = "space"
docstring-code-format = true
skip-magic-trailing-comma = true
docstring-code-line-length = "dynamic"


[tool.ruff.lint.pydocstyle]
convention = "pep257"

[tool.pyright]
exclude = [".github", "**/__pycache__", "**/data", "**/migrations"]
stubPath = "./.stubs"
venvPath = "./.venv"
venv = "./.venv/bin/python"

And I am currently running uv as the package manager. Ruff does not say anything about imports, so I am wondering why they have an issue.

p.d If I delete the pyright config the import error goes away but then I have the memory and vice versa.

[heejaechang]

you can control how things are indexed in multiple ways. you can use exclude to control what to include in your workspace. but that will impact everything. if you want to only control indexing, you have 3 options you can set in settings.json (not in pyproject.toml since the option is only for IDE)

python.analysis.indexing, python.analysis.packageIndexDepths, python.analysis.userFileIndexingLimit

see https://github.com/microsoft/pylance-release/blob/main/README.md#settings-and-customization for more detail.