How to rebuild the _ext.cpp when developing?

[kawaho]

Sorry this is a really basic question. I am trying to add some features into _ext.cpp. How do I rebuild the package after modifying the file?

In the README.md, it says

bash python -m pip install '.[test]'

but I assume this is just for the first time building the package. I imagine there is a way to only recompile the changes I did in the .cpp files?

[matthewfeickert]

I am trying to add some features into _ext.cpp.

@kawaho Please open up an Issue (https://github.com/scikit-hep/fastjet/issues) that describes the features that you want to add, so these can get discussed in advance.

How do I rebuild the package after modifying the file?

Building happens during the install, so just install your modified version with

python -m pip install --upgrade .

[kawaho]

Hi @matthewfeickert , coming back to this question, I was not able to rebuild the package using

python -m pip install --upgrade .

When I do so, I get the following error:

  building 'fastjet._ext' extension
  /isilon/export/home/kho29/mambaforge/envs/fastjet_dev/bin/x86_64-conda-linux-gnu-c++ -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /isilon/export/home/kho29/mambaforge/envs/fastjet_dev/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /isilon/export/home/kho29/mambaforge/envs/fastjet_dev/include -fPIC -I/isilon/export/home/kho29/fastjet_dev/fastjet/src/fastjet/_fastjet_core/include -I/tmp/pip-build-env-1urvtoly/overlay/lib/python3.12/site-packages/pybind11/include -I/isilon/export/home/kho29/mambaforge/envs/fastjet_dev/include/python3.12 -c src/_ext.cpp -o build/temp.linux-x86_64-cpython-312/src/_ext.o -fvisibility=hidden -g0 -std=c++11
  /isilon/export/home/kho29/mambaforge/envs/fastjet_dev/bin/x86_64-conda-linux-gnu-c++ -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /isilon/export/home/kho29/mambaforge/envs/fastjet_dev/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /isilon/export/home/kho29/mambaforge/envs/fastjet_dev/include -pthread -B /isilon/export/home/kho29/mambaforge/envs/fastjet_dev/compiler_compat -shared -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/isilon/export/home/kho29/mambaforge/envs/fastjet_dev/lib -Wl,-rpath-link,/isilon/export/home/kho29/mambaforge/envs/fastjet_dev/lib -L/isilon/export/home/kho29/mambaforge/envs/fastjet_dev/lib -fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /isilon/export/home/kho29/mambaforge/envs/fastjet_dev/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /isilon/export/home/kho29/mambaforge/envs/fastjet_dev/include build/temp.linux-x86_64-cpython-312/src/_ext.o -L/isilon/export/home/kho29/fastjet_dev/fastjet/src/fastjet/_fastjet_core/lib -Wl,-R$ORIGIN/_fastjet_core/lib -lfastjet -lfastjettools -lsiscone -lsiscone_spherical -lfastjetplugins -lfastjetcontribfragile -o build/lib.linux-x86_64-cpython-312/fastjet/_ext.cpython-312-x86_64-linux-gnu.so
  installing to build/bdist.linux-x86_64/wheel
  running install
  error: [Errno 17] File exists: 'build/lib.linux-x86_64-cpython-312/fastjet/_fastjet_core'
  error: subprocess-exited-with-error
  
  × Building wheel for fastjet (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /isilon/export/home/kho29/mambaforge/envs/fastjet_dev/bin/python /isilon/export/home/kho29/mambaforge/envs/fastjet_dev/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmpgrzugkax
  cwd: /isilon/export/home/kho29/fastjet_dev/fastjet
  Building wheel for fastjet (pyproject.toml) ... error
  ERROR: Failed building wheel for fastjet
Failed to build fastjet
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (fastjet)

If I manually remove build and src/fastjet/_fastjet_core/, everything gets built from scratch instead of just updating my modifications. Any suggestions on this? Thanks!

[kawaho]

Hi @matthewfeickert ,

You should not be removing the FastJet C++ source code. This is definitely not correct.

This is the part where I got confused because there are fastjet-core/ and src/fastjet/_fastjet_core. The latter is created after the build so I thought it's ok to remove it as pip complains something like src/fastjet/_fastjet_core already exists .

Please explain what it is you're actually trying to do and what the state of the repository is in. Though again, if you want to contribute additions to the source code open an Issue first so that there can be a discussion in advance.

Yes, I have opened an issue some time ago: #303

[matthewfeickert]

The latter is created after the build so I thought it's ok to remove it as pip complains something like src/fastjet/_fastjet_core already exists .

@kawaho My apologies, you're absolutely right, and even though you pointed everything out clearly and I said I carefully reread things I still misread the paths. Sorry.

You're correct that do do rebuilds you would need to remove this transient directory. This, again, is not something that you as a developer should be responsible for manually having to keep track of, so we again see the need for Issue #279 and Issue #310 to get dealt with.

[matthewfeickert]

I've opened PR #311 to partially address parts of this.

[matthewfeickert]

Between PR #311 and #279 (comment) I think this is now fully resolved so that one can now just run

python -m pip install --upgrade .

from the top level directory and get a working rebuild. Until Issue #279 is fully resolved, to get a totally clean slate for all of the Git submodules you can run #279 (comment) in advance of install.