Reuse virtualenvs across different sessions

[agravier]

Hello. I am looking at accelerating my nox runtimes on some CI server. In fact, I was looking at accelerating my tox runtimes until yesterday, but what I tried to achieve eluded me, and I had a hard time understanding what I could and could not ultimately do with the tox config file. I decided to try a more programmatic approach with nox. It suits me very much, so I'd like to stick with nox anyway.

So, the approach consists in reusing my venv across nox sessions where I know I can safely do so. I do not mean to keep each session's venv around across runs and reuse it (this is already a feature of nox afaiu but it won't happen on the CI server anyway), but to use the same venv for two different sessions in the same run.

How would this feature be useful?
I want my nox suite to run as quickly as possible. As the dev, I know where it's safe to reuse a virtualenv. So I'd like to be able to specify which sessions can share a venv.

Describe the solution you'd like
There are several approaches, some more flexible than others. Here is something that I think doesn't quite align with the idea behind the design of nox but aligns with my hackish attempt at solving the problem illustrated below. I understand that parametrized sessions make distinct venvs. We could mark some parameters as "not venv-sensitive" in the code below: @nox.parametrize('operation', ['bla', 'blo'], different_venv=False)`.
Now, the more flexible solution IMO, and one that may align with the design of nox (?), consists in explicitly allowing the user to define (at run-time) an "environment key" and make nox use it to fetch an existing env or create a new one.

@nox.session(
    python=['3.7', '3.8'])
@nox.parametrize('operation', ['lo', 'behold'])
def thing1(session: nox.sessions.Session, operation: str):
    session.use_venv(f'default-{operation}')
    ...


@nox.session(
    python=['3.7'])
def thing2(session: nox.sessions.Session, operation: str):
    session.use_venv('default-behold')
    ...

So, while there are 5 sessions defined, 4 env can potentially be created:

• "default-lo-3.7"
• "default-behold-3.7", reused by thing2-3.7
• "default-lo-3.8"
• "default-behold-3.8"

Describe alternatives you've considered
I've tried this:

from enum import Enum
from typing import Callable

import nox
import nox.sessions
import nox.virtualenv


def install_requirements_and_package(session: nox.sessions.Session):
    session.install('-r', 'requirements-dev.txt')
    session.install('.[all]')


class OpCode(Enum):
    check = 'check'
    unit = 'unit'
    functional = 'functional'
    performance = 'performance'


class Op:
    def __init__(self, session: nox.sessions.Session):
        self.session = session

    def __getitem__(self, item: OpCode) -> Callable[[], None]:
        return getattr(self, item.value)

    def check(self):
        print('duh-check')

    def unit(self):
        print('duh-unit')

    def functional(self):
        print('duh-functional')

    def performance(self):
        print('duh-performance')


@nox.session(
    python=['3.7', '3.8'],
    reuse_venv=True)
@nox.parametrize('operation', [op.value for op in OpCode])
def run(session: nox.sessions.Session, operation: str):
    opcode: OpCode = OpCode[operation]
    op = Op(session)[opcode]
    install_requirements_and_package(session)
    op()

It didn't work, of course. Which makes sense. Parametrized sessions end up listed as different entries in nox --list. I guess I hoped that nox would just create a venv per interpreter.

But I anyway doubt that this is the right way to specify which sessions can share an env.

[agravier]

@theacodes I want to work on that particular approach (the one I suggest is the right one, with session.use_venv), but I would like to do so once you give me some sign that this is in line with the design philosophy for nox. Could you please let me know if it is? Thanks.

[theacodes]

Hrm. I don't totally love the idea of this feature, tbh. But...

I think I would prefer us to be allow users to specify the name of the virtualenv and optionally set that in a way that intentionally causes collisions, so maybe something like:

@nox.session(..., venv_name="something")
# Uses "something" as the venv name seed, generates "something-3.8", etc."

Whereas this intentionally causes name collisions so that they'll end up re-using the venv:

def venv_namer(python, *args, **kwargs):
    return "non-unique-name"

@nox.session(..., venv_name_func=venv_namer)

[agravier]

Works for me. I'll come up with a PR or come back here for more discussion if I hit other design questions.

[funkyfuture]

this seems to be the same issue as #167 to me.

[smarie]

This is a great topic. I would love to be able to

• reuse environments across multiple sessions
• mark a session as dependent on other sessions to be executed first

However these two concepts should remain independent in my opinion (which is not the case in the related proposal #167 ). I can create a session B dependent on session A being executed first (and creating some results files) while each session having its own environment.

I think that a great way to achieve this would be to have shared environments would be to have the optional possibility to explicitly declare the environments separately. I prefer such design where "shared env" is explicit, rather than implicit by mixing the notion of session dependency and env reuse.

@nox.venv(python=["2.7", "3.5"], venv_backend="conda")
def my_shared_env(env):
    # shared installation steps: `env` behave like a `session`
    env.conda_install(...)

@nox.session(venv=my_shared_env)
def do_a(session):
    # session.conda_install would raise an error here "this session uses a shared environment !"
    ...

@nox.session(venv=my_shared_env)
def do_b(session):
    # session.conda_install would raise an error here "this session uses a shared environment !"
    ...

@nox.session(python="3.8", requires=do_a) 
def do_c(session):
    # note that do_c does not reuse my_shared_env even if it depends on session do_a to be executed first
    # so it would run in a dedicated 3.8 python while do_a would run on my_shared_env
    ...

My two cents...

[theacodes]

Closed in favor of #167