re-using virtual environment (Linux)

[kakila]

Thank you all for the fantastic work!

I am finding integrating firedrake to our group workflow slightly difficult.
We execute many projects per year and those that use python, and for the sake of reproducibility, each specify they requirements independently.
A new person working on the project, would clone the corresponding repository and create a local virtual environment using the given requirements.
For firedrake we provide a special script (basically get the install script and install).
Most of the python projects extend the functionality in the python virtual environment. That is they install other python tools on top of what firedrake already installs.
The issue is the following: if the user already has another project already using firedrake, how can they reuse the already existing virtual environment?

The naive answer is to just use the existing virtual env. However, if the new project installs more python dependencies or has version conflicts with other projects using the same firedrake virtual env, problems arise (firedrake arguments agaisnt Conda for a similar reason: it doesn't isolates well). In the former case the firedrake virtual env gets polluted (and updates become increasingly expensive), in the latter case the projects working env breaks.

A simple example

1. create firedrake install with option --venv-name project1
2. create another firedraek install with option --venv-name project2

project2 pays the whole price and doesn't resuse builds from project1

Ideal workflow

1. create a firedrake install with option --venv-name firedrake (the intention is that everyone can use this install)
2. create virtual environment my_project using the dependencies of that project
3. "merge" the virtual envs to get firedrake running on my_project

So the questions :

• Is it possible to copy over (or link) the expensive parts of a firedrake vritual env install (e.g. petsc)? Are there instructions on how to do this properly?
• Is it better to have a system install of the expensive parts and make sure a new firedrake installation uses those? (without rebuilding)

Maybe you have other ideas/solutions!
Thanks!

[connorjward]

The best solution for this is to have a single build of PETSc that can be reused between venvs. To do this you need to:

1. Run python3 firedrake-install --show-petsc-configure-options to get the list of PETSc options that we need.
2. Install PETSc

$ git clone https://github.com/firedrakeproject/petsc.git
$ cd petsc
$ ./configure  # using the options you found above
$ make

3. export PETSC_DIR=where/petsc/is, export PETSC_ARCH=something.
4. python3 firedrake-install --honour-petsc-dir (will use the existing PETSc).

[kakila]

Thanks a lot, looking at the virtual environment created this way (did not test yet), I see it has a size of about 1.9 GB instead of 6.4 GB. This is a huge improvement in light-weightedness! 👍
Is it possible to reduce this further to say about 0.5 GB? The next targets would be
75M src/pyadjoint
94M src/h5py
123M src/libsupermesh
294M src/firedrake
4.5G src/petsc (done!)

Anyway of pre-building and sharing any of those across virtual envs?

[dham]

We are pushing towards having pip install work. At that stage users won't have the source repositories lying around after installation. This should significantly reduce the size of the install. See: https://www.firedrakeproject.org/download.html#installing-firedrake-with-pip-experimental-linux-only

With the exception of libsupermesh, those are Python packages, so sharing between venvs won't really work, but the installed packages might then be small enough that this matters less.

[kakila]

I see, so the only further reduction will be supermesh. I think this is good enough.
Thanks!

[dham]

No, the installed packages will be a lot smaller than the current git repositories. E.g. most of the core firedrake package is the documentation folder and its git archive. I think a pip installed firedrake package will be on the order of 20MB rather than 300MB.

[wence-]

Just to note as well once those pip packages are available, if you use uv rather than pip to do the installs, the actual packages (if everything is on the same filesystem) will be hardlinked into each venv in turn (rather than each venv being separate)