Can we help more of our community understand and use virtual environments?

[hsu000001]

Who uses virtual environments? At a previous CSDMS Annual Meeting, the Terrestrial Working Group discussion, the topic of virtual environments came us as one we'd like to learn more about!

Example real conversation:
Person 1: I've heard about virtual environments, but what's the difference between that and regular environments?
Person 2: Mark and Eric are always telling me to use virtual environments and how useful they are. But I just use my standard python environment.
Person 1: Yeah, me too, I just pip install stuff when it tells me to!
Person 2: Me too!!

Take the poll below to let us know if you are interested in learning more, and leave a comment if you have a link or tip to help us collectively use virtual environments more efficiently.

We are interested to hear:

• what are they?
• why should I use them?
• what's the simplest way to get started?
• how can I change my workflow to start using them?
• will I be perceived as really cool if I use them?

[hsu000001]

Here are some excerpts from another thread I observed. Personally I've only tried venv and need very explicit instructions of how to do so...
Out of curiosity, what python virtual environment managers/tools are people using these days? conda, mamba, poetry, pyenv, venv, pipx, others?

• I was recommended pyenv paired with pyenv-virtualenv
• For scientific projects (or Windows), conda has worked quite well for me. This seems to be the most popular env manager among the scientists I've worked with, and it is simple enough to use in Docker containers. And, if you use containers, conda-pack is very helpful when trying to minimize layers and keep things small.
• I also like pip-compile in the pip-tools package for transferring environments or building the environment from a requirements.txt file in an image's Dockerfile.
• After seeing too many issues getting conda envs to resolve, I switched to pyenv + pipenv. I gave poetry a try at some point since it's well-liked by the python community, but for some reason that's lost to time, I still preferred pipenv.
• I use a similar setup with poetry and pipx. My decision on poetry vs pipenv was arbitrary as both would suit my use cases; I just wanted something more powerful than regular pip and venv. pyenv is a cool tool to have, but it's another I haven't had much use for yet.
• I use mamba for scientific projects (caveat: I haven't given the other tools a fair shot).
• venv, conda, pipenv, poetry, venv
• pyenv is great if you need multiple versions of Python on your system at once. It allows me to have Python 3.8, 3,9, and 3.10 accessible. I can set one of them to be global and if needed, I can set one to be used in a shell environment via pyenv shell 3.8.16 (or whatever).

[hsu000001]

Actually all of the answers to the poll should be "I have not heard of these" since I have a typo in "virtual." I have not heard of vitual environments. I cannot fix it unless I reset the poll, so this typo will just persist.

[samharrison7]

Thanks for starting this thread! Virtual environments are super important, but a real minefield.

What are they?

This is a really good blog post explaining the Python situation: https://alpopkes.com/posts/python/packaging_tools/. It nicely explains why there are so many different virtual environment (+ package management, depedency management, package building, etc...) tools out there and what their strengths are. It's about much more than virtual environments, but I kind of feel you have to touch on the other aspects to get a full understanding of what virtual environments are all about.

The only thing that post misses is that there is a new package manager on the block that solves a lot of the issues around Conda. It's called Pixi: https://pixi.sh/latest/ (developed by the folks that developed mamba). It kind of takes the best bits of Conda and pip and brings them together, except they've re-written everything to be super fast. I honestly think Pixi will be the future of package management.

What's the simplest way to get started?
How can I change my workflow to start using them?

It's personal preference, but my advice would be to pick a environment manager (maybe based on that blog I linked to, or my random thoughts below) and dive in. Every time you start a new project (i.e. a new bit of code), force yourself to use the environment manager to create a new virtual environment. Treat them as ephemeral - don't be afraid of deleting them and re-creating them on a whim. That's the whole beauty of them. Soon you'll be wondering how you ever survived without them.

Will I be perceived as really cool if I use them?

Yes 😎

A few other random thoughts:

• The strengths to Conda/Mamba/Pixi is that they're not exclusive to Python packages. You can theoretically create a Conda package written in any language. I often mix up Python and Fortran in my projects and Conda is really useful for this.
• If you're currently using Conda, then swap it for Mamba. Mamba is a drop-in replacement (as in, you write the same commands but with mamba instead of conda), but is much quicker and more reliable at resolving dependencies than Conda.
• If you're developing pure Python packages, Poetry is wonderful, but is perhaps a bit more of a steep learning curve than pyenv etc.
• What do I use? Increasingly Pixi, previously Mamba, occassionally Poetry if I'm dealing with only Python.