Instructions to create a python virtual environment

[dbeatty10]

As an end user, you typically want to do installations within an isolated environment so-as to not affect the rest of your system.

There are a variety of tools within the Python ecosystem that empower isolated environments (conda, poetry, venv, etc). The option with the least additional dependencies is venv which has been automatically available with the latest versions of Python for quite a while.

We should document how to create and use venv virtual environments to make it easier for people to perform install steps.

This is relevant anytime a user is needs to run dbt locally (e.g., using outside of dbt Cloud or some other remote executor).

Example instructions from here:

A couple typical places to see instructions like this:

• CONTRIBUTING.md within dbt Core + adapters
• dbt projects

Maybe related to: #1235

[lostmygithubaccount]

I'm wary about taking on the responsibility of documenting this (given the number of options and platform-specific considerations), but +1 for venv. I also think we should clearly give some explanation of what's going on:

• venv creates a virtual Python environemnt -- that is, a directory with a Python interpreter + the libraries you install in that environment
• -m there is calling the built-in module named venv after invoking Python
• I'd recommend not calling the venv name venv to avoid confusion there -- either my_env or dbt_env or something, and clearly state this is the name
• what happens now when you pip install things with the env active (show that those files go in the directory created earlier)
• turning it on/off (what is source doing? can you run that on Windows?)
• best practices (source control, where to store venvs)

[dbeatty10]

Yeah, makes sense to avoid name collision with venv with a different name like env, my_env, etc.

Agreed to the default position of not taking on the responsibility of documenting things like this.

But here's why we should consider it strongly anyways:

• it's a fundamental entry point for dbt contributors, but yet not described in a simple and hyperlink-able manner by a trusted source
  • these instructions are not very approachable or user-friendly 😢
  • since venv is very configurable, docs.python.org doesn't offer opinionated commands like python3 -m pip install --upgrade pip
• we can validate platform-specific in CI like this
• as a user, it's not fun to be left to your own imagination how best to do this, and the internet will offer an overwhelming number of opinions and options
  • much easier if we can provide one set of instructions (among multiple options) that is vetted and will get them from A to B.

[dbeatty10]

dbt Labs already includes some venv-specific instructions in these places:

• https://docs.getdbt.com/dbt-cli/install/from-source#installing-adapter-plugins
  • click link labeled "What are the best practices for installing dbt Core with pip?"
• https://docs.getdbt.com/reference/warehouse-profiles/glue-profile#configuration-of-the-local-environment
• https://docs.getdbt.com/reference/warehouse-profiles/materialize-profile#installation-and-distribution

Exhortations to use virtual environments (but without further instructions or links):

• https://docs.getdbt.com/dbt-cli/install/pip
  • "We recommend using virtual environments when installing with pip"
• https://docs.getdbt.com/docs/contributing/adapter-development/4-testing-a-new-adapter#install-dependencies
  • "You should already have a virtual environment with dbt-core and your adapter plugin installed"
• https://docs.getdbt.com/blog/open-source-community-growth-analysis#configuring-openlineage
  • "hey, folks, always use a virtual environment!)"
• https://docs.getdbt.com/docs/running-a-dbt-project/running-dbt-in-production#airflow
  • "In this case, be sure to install dbt into a virtual environment to avoid issues with conflicting dependencies between Airflow and dbt"

This blog post included an external link to https://docs.python.org/3/library/venv.html:

• https://docs.getdbt.com/blog/enforcing-rules-pre-commit-dbt#activating-a-python-virtual-environment