cadabra2 fails to build with Python 3.12

[doko42]

see https://bugs.debian.org/1061319

we currently have both 3.11 and 3.12 installed during the build, and cadabra2 picks up 3.11, ignores 3.12. is there a way to use a specific Python version?

[kpeeters]

@doko42 This is related to pybind11 which is included in the source tree, and picks up the wrong python. Can you check release 2.4.5.4 (I just tagged & released one on github) and let me know? That one includes a more recent pybind11.

[doko42]

no, still selects 3.11:

---

Configuring Python

-- Building for use with Python 3 (good!)
-- pybind11 v2.12.0 dev1
CMake Warning (dev) at cmake/modules/FindPythonLibsNew.cmake:60 (find_package):
Policy CMP0148 is not set: The FindPythonInterp and FindPythonLibs modules
are removed. Run "cmake --help-policy CMP0148" for policy details. Use
the cmake_policy command to set the policy and suppress this warning.

Call Stack (most recent call first):
libs/pybind11/tools/pybind11Tools.cmake:50 (find_package)
libs/pybind11/tools/pybind11Common.cmake:188 (include)
libs/pybind11/CMakeLists.txt:211 (include)
This warning is for project developers. Use -Wno-dev to suppress it.

-- Found PythonInterp: /usr/bin/python3.11 (found version "3.11.7")
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.11.so
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Found python /usr/lib/x86_64-linux-gnu/libpython3.11.so
-- Python version is 3.11.
-- Installing Python module in /usr/local/lib/python3.11/dist-packages
-- Python abi name cpython-311-x86_64-linux-gnu
-- Python module extension cpython-311-x86_64-linux-gnu.so

[kpeeters]

Ah, it's picking up a duplicate FindPythonLibsNew.cmake, instead of using the one from pybind11. I have removed it and pushed to github. Can you test against current head or do you need a tagged release?

[doko42]

that works: https://launchpad.net/ubuntu/+source/cadabra2/2.4.5.4-0ubuntu1

[kpeeters]

Ok. That package still misses a number of dependencies in order to run correctly, in particular texlive. Also, the package that is currently in the repos for 22.04 fails to start because it refers to /usr/lib/python3.10/site-packages/cadabra2 which does not exist (I did not check whether the update for noble has that problem too). Am happy to help to get this to work properly, but I am not really up to date wrt. current debian guidelines as to what should go where, so please let me know how I can help.

[alexmyczko]

@kpeeters sorry for the delay, but i'll try the latest version on debian now...
builds but need some more work (tries to use 3.11): http://bananas.debian.net/debian/cadabra2/

a little bit over complicated, basically i have these:

config/Doxyfile:                         core/cadabra2_defaults.py \
client_server/Server.cc:	//	pybind11::eval_file(PYTHON_SITE_PATH"/cadabra2_defaults.py");
client_server/Server.cc:	   "f=open(r'" + python_path + "/cadabra2_defaults.py'); "
client_server/Server.cc:	   "code=compile(f.read(), 'cadabra2_defaults.py', 'exec'); "
debian/clean:core/cadabra2_defaults.py
jupyterkernel/cadabra2_jupyter/context.py:        with open(os.path.join(SITE_PATH, "cadabra2_defaults.py")) as f:
jupyterkernel/cadabra2_jupyter/context.py:            code = compile(f.read(), "cadabra2_defaults.py", "exec")
doc/modules.dox:/// parts. One is core/cadabra2_defaults.py, which is a pure-python file.
grep: doc/thesis_dominic_price.pdf: binary file matches
core/cadabra2.in:        filepath = os.path.join(d, "cadabra2_defaults.py")
core/cadabra2.in:                code = compile(f.read(), "cadabra2_defaults.py", 'exec')
core/cadabra2.in:        sh.runsource("import os; import sys; install_prefix=os.path.realpath(sys.argv[0]); install_prefix=install_prefix.replace('/bin/cadabra2','${PYTHON_SITE_PATH_REL}'); sys.path.append(install_prefix); print('Cadabra @CADABRA_VERSION_MAJOR@.@CADABRA_VERSION_MINOR@.@CADABRA_VERSION_PATCH@ (build @CADABRA_VERSION_BUILD@ dated @CADABRA_VERSION_DATE@)'); print ('Copyright (C) @COPYRIGHT_YEARS@  Kasper Peeters <kasper.peeters@phi-sci.com>'); f=open('${PYTHON_SITE_PATH}/cadabra2_defaults.py'); code=compile(f.read(), 'cadabra2_defaults.py', 'exec'); exec(code); f.close(); print('Using SymPy version '+sympy.__version__);")
core/cadabra2_defaults.py.in:# \file    cadabra2_defaults.py
core/CMakeLists.txt:   "${PROJECT_SOURCE_DIR}/cadabra2_defaults.py.in"
core/CMakeLists.txt:   "${PROJECT_SOURCE_DIR}/cadabra2_defaults.py"
core/CMakeLists.txt:   # remove_file("${OLD_PYTHON_SITE_PATH}/cadabra2_defaults.py")
core/CMakeLists.txt:   # remove_file("${OLDER_PYTHON_SITE_PATH}/cadabra2_defaults.py")
core/CMakeLists.txt:      "${PROJECT_SOURCE_DIR}/cadabra2_defaults.py" 
core/.gitignore:cadabra2_defaults.py
core/CdbPython.cc:	//	    << "   code = compile(f.read(), 'cadabra2_defaults.py', 'exec')\n"
core/CdbPython.cc:// //	    << "   code = compile(f.read(), 'cadabra2_defaults.py', 'exec')\n"
core/cadabra2-cli.cc:	// Run cadabra2_defaults.py
core/cadabra2-cli.cc:		execute_file(site_path + "/cadabra2_defaults.py", false);

and i want just one place to set it to /usr/lib/python3/dist-packages/cadabra2_defaults.py
maybe better /usr/lib/python3/dist-packages/cadabra2/cadabra2_defaults.py

[kpeeters]

Sorry, github somehow didn't notify me about the edit you made to this comment, so I missed the key part. I'll have a look today. Meanwhile, does the current master branch still insist on building with 3.11 for you?

[alexmyczko]

yes and i want to have a single like configuration for cadabra2_defaults.py path and not 20 random lines distributed across all the files generating it in some magical way. the latest two issues also affect me

[kpeeters]

(Welcome to the mess that is Python package distribution... No matter what I do, there is always some system that decides that they want to do it differently.)

Just to be clear: unless there is a bug, and unless you are inside an AppImage (which doesn't apply to you), then cadabra should only ever try to load cadabra2_defaults.py from PYTHON_SITE_PATH, which you can set at configuration stage using

cmake -DPYTHON_SITE_PATH=...

Is there anything you have found that does not stick to this promise? If so then of course I'm happy to fix that.

(I agree that PYTHON_SITE_PATH/cadabra2/ would have been a better choice, but if I change that now, we will have updates-from-source going haywire on loads of installations. So unless there is a very good reason to change this, I'd prefer not to).