synapse fails to install with poetry 1.6.0 - "No module named 'synapse.synapse_rust'" at runtime

[michaelkaye]

Description

originally reported in: michaelkaye/setup-matrix-synapse#95

This was last known working on 18 Aug at 17:00 UTC

No error seems to occur when poetry install is run.

Steps to reproduce

Steps we do for install are written in the GHA action: https://github.com/michaelkaye/setup-matrix-synapse/blob/main/create.js#L18

See github action results for the error: https://github.com/michaelkaye/setup-matrix-synapse/actions/runs/5924305224

Homeserver

testing server in GHA

Synapse Version

main

Installation Method

Other (please mention below)

Database

n/a - can't start synapse

Workers

Single process

Platform

Installed via michaelkaye/setup-matrix-synapse in a GHA worker

Configuration

N/A - can't start synapse

Relevant log output

/opt/pipx_bin/poetry run python -m synapse.app.homeserver --server-name localhost --config-path homeserver.yaml --generate-config --report-stats=no
Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 112, in _get_module_details
  File "/home/runner/work/setup-matrix-synapse/setup-matrix-synapse/synapse/synapse/__init__.py", line 24, in <module>
    from synapse.util.rust import check_rust_lib_up_to_date
  File "/home/runner/work/setup-matrix-synapse/setup-matrix-synapse/synapse/synapse/util/rust.py", line 20, in <module>
    from synapse.synapse_rust import get_rust_file_digest
ModuleNotFoundError: No module named 'synapse.synapse_rust'
Error: The process '/opt/pipx_bin/poetry' failed with exit code 1

Anything else that would be useful to know?

Installing via pip works fine - see alternate test cases in the github action.

[DMRobertson]

Can you try passing -vv to poetry install and seeing if that spews out any error context? It should generate a file called synapse_rust.abi3.so---suggest getting the action to do a find . -name '*.so' to confirm this exists in a sensible location. (For me this gets generated in the synapse directory (that containing e.g. py.typed).)

[DMRobertson]

This was last known working on 18 Aug at 17:00 UTC

Hmm. Poetry did a release over the weekend. Do you pin the version of poetry that you use?

[DMRobertson]

Synapse uses 1.3.2, e.g.

synapse/.github/workflows/tests.yml

Line 43 in 561d06b

poetry-version: "1.3.2"

Though I would expect more recent versions to work as well.

[michaelkaye]

Thanks for the suggestions.

This is reproducible locally with the same error; i can't see the generated .so file mentioned above on my desktop.

We haven't pinned the version of poetry; locally i'm using poetry 1.6.0

Going to try:

• with the verbose flags on the GHA now.
• with poetry pinned to 1.3.2 on the GHA now.

[michaelkaye]

poetry -vv doesn't output any more information: (perhaps we're actually hiding a lack of errors somewhere) https://github.com/michaelkaye/setup-matrix-synapse/actions/runs/5925091505/job/16063832535#step:4:69

poetry 1.3.2 works fine: https://github.com/michaelkaye/setup-matrix-synapse/actions/runs/5925090308/job/16063828347

Going to pin to latest poetry before 1.6.0 to see if that helps, but maybe something in that latest release that's causing issues then.

[michaelkaye]

Even with -vvv the only extra messages I get in GHA are as follows, and checking for the .so file finds nothing:

 Installing the current project: matrix-synapse (1.90.0)
  - Building package matrix-synapse in editable mode
  - Falling back on using a setup.py

see https://github.com/michaelkaye/setup-matrix-synapse/actions/runs/5925479981/job/16064987664

Pinning poetry to 1.5.1 has solved my personal problem here; let me know if there's anything else debugging wise i can do to test

I tell a lie: The .so file exists:

https://github.com/michaelkaye/setup-matrix-synapse/actions/runs/5925479981/job/16064987664#step:5:1115

[clokep]

We've gotten a few other reports of this biting people now; we might need to figure this out.

[clokep]

With poetry 1.6.1 I end up with a .so file in my build dir

$ find . -name '*.so'
./build/lib.macosx-13.3-x86_64-cpython-311/synapse/synapse_rust.abi3.so

If I move this into my Synapse directory it works fine: mv ./build/lib.macosx-13.3-x86_64-cpython-311/synapse/synapse_rust.abi3.so synapse

[clokep]

We're fairly certain a new release of poetry-core will fix this, probably due to python-poetry/poetry-core#633.

diff --git a/pyproject.toml b/pyproject.toml
index 5fb64479a1..23ef74885e 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -381,7 +381,7 @@ furo = ">=2022.12.7,<2024.0.0"
 # system changes.
 # We are happy to raise these upper bounds upon request,
 # provided we check that it's safe to do so (i.e. that CI passes).
-requires = ["poetry-core>=1.1.0,<=1.7.0", "setuptools_rust>=1.3,<=1.7.0"]
+requires = ["poetry-core @ git+https://github.com/python-poetry/poetry-core.git#main", "setuptools_rust>=1.3,<=1.7.0"]
 build-backend = "poetry.core.masonry.api"
 

[thebalaa]

The root of this issue, IMHO, is the build configuration of the synapse.synapse_rust module. I suggest renaming the module to not be namespaced under the top-level synapse module.

The issue appears to crop up when doing editable installs of this repo with poetry install --extras all or pip install -e . AND attempting to import the synapse module from code while your CWD is the root of this repo.

Manually copying the .so file from build/**/synapse_rust.abi3.so to ./synapse resolves the issue, perhaps updating the rust module build configuration to do this for editable installs is another possible solution.

At the time of this writing there is no longer an issue with Poetry not properly installing editable modules with extensions, re: python-poetry/poetry-core#633

I have tested with the latest version of poetry 1.7.1

On my system this is the trick to resolve the issue:

cp build/lib.linux-x86_64-cpython-311/synapse/synapse_rust.abi3.so synapse/

Unfortunately many tumultuous hours were spent tracking this down. This issue is a roadblock for new contributors making contributions to Synapse.

[clokep]

Manually copying the .so file from build/**/synapse_rust.abi3.so to ./synapse resolves the issue, perhaps updating the rust module build configuration to do this for editable installs is another possible solution.

I'm very unconvinced that if we left the module name as just synapse_rust then it would be moved properly -- did you test this?

Unfortunately many tumultuous hours were spent tracking this down. This issue is a roadblock for new contributors making contributions to Synapse.

Yes, it is very frustrating, but we do recommend using 1.5.x in the docs, I don't think there's anything else we can do, unfortunately.

[DMRobertson]

@thebalaa: thanks for taking a look at this. We know this issue causes pain for contributors---it does for us too---and we would love to understand the right way to fix this. Python packaging is not fun, and by using poetry + poetry's build script + pyO3 + setuptools_rust we have certainly managed to make life hard for ourselves. Any help you can provide would be gratefully received.

To summarise my understanding of the situation (please double-check my reasoning):

• Each poetry install ensures a shared object is built in build.
• When you're bitten by this issue, poetry install does not copy that shared object to the synapse directory.
  • We only noticed this behaviour ourselves on Poetry 1.6; as synapse fails to install with poetry 1.6.0 - "No module named 'synapse.synapse_rust'" at runtime #16147 (comment) notes, Poetry 1.5.1 seemed to copy across the .so fine and not hit this problem.
• In this situation, attempts to import synapse fail because synapse.synapse_rust isn't available to the import machinery.

The root of this issue, IMHO, is the build configuration of the synapse.synapse_rust module. I suggest renaming the module to not be namespaced under the top-level synapse module.

I don't follow how this would help. Suppose we changed the config to produce a shared object foo.so which provided a python module called bar, and made Synapse import bar. If you're still affected by this problem, then the shared object won't get getting copied outside of the build directory---and then I don't see how the import machinery will find it. Indeed, on my machine, the build directory isn't part of my interpreter import path:

$ echo $PYTHONPATH


$ python -c "import sys; print(sys.path)"
['', '/usr/lib64/python311.zip', '/usr/lib64/python3.11', '/usr/lib64/python3.11/lib-dynload', '/home/dmr/.cache/pypoetry/virtualenvs/matrix-synapse-YNqvS2_4-py3.11/lib64/python3.11/site-packages', '/home/dmr/.cache/pypoetry/virtualenvs/matrix-synapse-YNqvS2_4-py3.11/lib/python3.11/site-packages', '/home/dmr/workspace/synapse']

$ find build -name *.so
build/lib.linux-x86_64-cpython-311/synapse/synapse_rust.abi3.so
build/lib.linux-x86_64-cpython-310/synapse/synapse_rust.abi3.so

(I'm very happy to be proven wrong---as Patrick says in #16147 (comment), if you can reproduce the problem and test to demonstrate this fix works then we'd be very interested!!)

[michaelkaye]

If you can give me some specific instructions to follow (or a branch of synapse to try to install, etc etc) I'm happy to use the reproducible failure from the synapse GHA to prove success/failure of this using various poetry versions.

I might swing by the channel tomorrow?

[thebalaa]

I was mistaken this is a poetry issue after all: python-poetry/poetry#8539

[dimbleby]

you will never know whether this is fixed in recent poetry-core, if you do not use recent poetry core

synapse/pyproject.toml

Line 385 in 825ac7e

requires = ["poetry-core>=1.1.0,<=1.7.0", "setuptools_rust>=1.3,<=1.8.1"]

[clokep]

you will never know whether this is fixed in recent poetry-core, if you do not use recent poetry core

We have been periodically trying updated poetry-core versions. We were really hoping 1.7.0 would fix it, but the issue seems to persist.

[dimbleby]

You knew in September that this was fixed in 1.8.0, I don't know when you forgot?

#16147 (comment)

[thebalaa]

This should fix it: #16702

[clokep]

You knew in September that this was fixed in 1.8.0, I don't know when you forgot?

#16147 (comment)

I must have misquoted a version number, IIRC installing from git didn't work though and I didn't have further time to debug it. Maybe now that there's a release things will work properly.

[michaelkaye]

So I've poked at this off and on today:

Moving to poetry 1.7.1 for the poetry run, (1.7.1 depends on poetry-core 1.8.1) on it's own, doesn't work.

We seem to need to both upgrade the poetry used to install and the one defined within synapse.

So poetry 1.7.1 + the patch in #16702 seems to work for getting this going.

See workflows from setup-matrix-synapse that are now completing with poetry versions > 1.5.x.

So basically I'd be in favour of incorporating this dependency bump.