Skip to content

Commit

Permalink
Merge pull request #376 from danykaufman/3.x_virtual_env
Browse files Browse the repository at this point in the history 
update virtual-env.ipynb documentation for new jupy
  • Loading branch information
@gilad-shaham
gilad-shaham authored Apr 19, 2023
2 parents a2cf04a + 2b20a5f commit def0cf7
Showing 1 changed file with 117 additions and 48 deletions.
165 changes: 117 additions & 48 deletions virtual-env.ipynb
View file
Original file line number Diff line number Diff line change
@@ -1,23 +1,28 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Creating Python Virtual Environments with conda"
"# Creating Python Virtual Environments with Conda"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"- [Overview](#conda-virual-env-overview)\n",
"- [Setting Up a Virtual Environment Using conda](#setting-up-a-virtual-environment-using-conda)\n",
" - [Creating a conda Virtual Environment from a File](#creating-a-conda-environment-from-a-file)\n",
"- [Setting Up a RAPIDS conda Environment with cuDF and cuML](#setting-up-a-rapids-conda-environment)"
"- [Preconfigured Conda environments](#preconfigured-conda-environments)\n",
"- [Setting Up a Virtual Environment Using Conda](#setting-up-a-virtual-environment-using-conda)\n",
" - [Creating a Conda Virtual Environment from a File](#creating-a-conda-environment-from-a-file)\n",
" - [Cloning an existing Conda Virtual Environment](#cloning-an-existing-conda-environment)\n",
"- [Setting Up a RAPIDS Conda Environment with cuDF and cuML](#setting-up-a-rapids-conda-environment)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
Expand All @@ -26,78 +31,116 @@
"\n",
"A Python virtual environment is a named, isolated, working copy of Python that maintains its own files, directories, and paths so that you can work with specific versions of libraries or Python itself without affecting other Python projects.\n",
"Virtual environments make it easy to cleanly separate projects and avoid problems with different dependencies and version requirements across components.\n",
"The conda command-line interface (CLI) is the preferred interface for managing installations and virtual environments with the Anaconda Python distribution.\n",
"The Jupyter Notebook service of the Iguazio Data Science Platform (\"the platform\") comes pre-deployed with conda.\n",
"This tutorial explains how to use conda to create a Python virtual environment that will be available as a custom kernel in Jupyter Notebook.\n",
"For general information about using conda to create virtual environments, see the [conda documentation](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html)."
"The Conda command-line interface (CLI) is the preferred interface for managing installations and virtual environments with the Anaconda Python distribution.\n",
"The Jupyter service of the Iguazio Data Science Platform (\"the platform\") comes pre-deployed with Conda.\n",
"This tutorial explains how to use Conda to create a Python virtual environment that will be available as a custom kernel in JupyterLab.\n",
"For general information about using Conda to create virtual environments, see the [Conda documentation](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html)."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"<a id=\"preconfigured-conda-environments\"></a>\n",
"## Preconfigured Conda environments\n",
"\n",
"The platform provides several preconfigured Conda environments that are included in the Jupyter pod:\n",
"- **base**: This is the default base environment that includes the Conda binaries. Do not use it for development purposes.\n",
"- **jupyter**: This environment incudes the JupyterLab server and all of its dependencies. Do not use it for development purposes.\n",
"- **mlrun-base**: This environment includes MLRun and all its dependencies preinstalled.\n",
"- **mlrun-extended**: This environment includes all the packages from mlrun-base, as well as additional packages such as TensorFlow, PyTorch, and scikit-learn, which are required for the demo notebooks.\n",
"\n",
"While the platform provides several preconfigured Conda environments that are included in the Jupyter pod, it's important to note that any Conda installations to these environments are not persistent. They are reset when the Jupyter service is restarted.\n",
"\n",
"However, when these Conda environments are activated, the `PIP_PREFIX` and `PYTHONPATH` environment variables are automatically set to the data mount directory of the running user: `/User/.pythonlibs/<environment name>`. This means that PIP installations persist even after restarting the Jupyter pod.\n",
"\n",
"Any new Conda environment that is created or cloned will be fully located in the data mount directory of the running user, specifically at `/User/.conda/<environment name>`. These environments are fully persistent."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"<a id=\"setting-up-a-virtual-environment-using-conda\"></a>\n",
"## Setting Up a Virtual Environment Using conda\n",
"## Setting Up a Virtual Environment Using Conda\n",
"\n",
"Follow these steps from your Jupyter Notebook service to create a Python virtual environment using conda: \n",
"Follow these steps from your Jupyter service to create a Python virtual environment using Conda: \n",
"\n",
"1. <a id=\"conda-virt-env-setup-step-create-terminal\"></a>Create a new terminal by selecting the **New Launcher** option (`+` icon) from the top action toolbar in the left sidebar, and then selecting **Terminal** from the main work area.<br>\n",
" The next steps should be executed from your new terminal, except where otherwise specified.\n",
"\n",
" The next steps should be executed from your new terminal, except where otherwise specified.<br>\n",
"\n",
"2. <a id=\"conda-virt-env-setup-step-create-env\"></a>Create a new Python virtual environment by running the following command.\n",
" Replace `<environment name>` with your preferred virtual-environment name:\n",
" ```sh\n",
" conda create -n <environment name> ipykernel python=3.7\n",
" conda create -n <environment name>\n",
" ```\n",
"\n",
" <br>For example, the following command creates an environment named \"myenv\":<br>\n",
" <br>For example, the following command creates an environment named \"myenv\":\n",
" ```sh\n",
" conda create -n myenv ipykernel python=3.7\n",
" conda create -n myenv\n",
" ```\n",
" <br>\n",
"\n",
"\n",
"3. <a id=\"conda-virt-env-setup-step-install-packages\"></a>Install the desired Python packages by running the following command.\n",
" Replace `<environment name>` with the name of the environment that you created in the previous step and `<package>` with the name of your selected package, optionally followed by `=<version>`; you can specify multiple packages:\n",
"3. <a id=\"conda-virt-env-setup-step-install-packages\"></a>Activate the Conda environment by running the following command:\n",
" ```sh\n",
" conda install -n <environment name> <package> [<package> ...]\n",
" conda activate <environment name>\n",
" ```\n",
"\n",
" <br>For example, the following command installs the SciPy, pandas version 0.25.0, and TensorFlow version 1.13.1 packages for a \"myenv\" environment:\n",
" \n",
" <br>For example, the following command activates an environment named \"myenv\":\n",
" ```sh\n",
" conda install -n myenv scipy pandas=0.25.0 tensorflow=1.13.1 python=3.7\n",
" ```\n",
" conda activate myenv\n",
" ``` \n",
" <br>\n",
"\n",
"4. <a id=\"conda-virt-env-setup-step-install-packages\"></a>Once you activate your environment, use either PIP or Conda to install the necessary Python packages.<br> To do this, replace `<package>` with the name of the package you want to install, and optionally add `==<version>` for PIP or `=<version>` for Conda. You can specify multiple packages in the same command.<br>\n",
"\n",
"4. <a id=\"conda-virt-env-setup-step-export-env\"></a>Export your new virtual environment to an environment file in a platform data container by running the following command.\n",
" Replace `<container name>` with the name of a platform data container, `<directory path>` with an optional relative container-directories path, and `<environment name>` with the name of the environment that you created:\n",
" ```sh\n",
" conda env export -n <environment name> > /v3io/<container name>[/<directory path>]/<environment name>.yaml\n",
" ```\n",
"\n",
" <br>It's recommended that you save the environment to a virtual-environments directory in your running-user home directory (**users/&lt;running user&gt;**).\n",
" For example, the following command creates a **users/&lt;running user&gt;/virtual_env/myenv.yaml** file:\n",
" ```sh\n",
" conda env export -n myenv > /v3io/users/$V3IO_USERNAME/virtual_env/myenv.yaml\n",
" ```\n",
" You can use the `User` data mount to the running-user directory to shorten this command (see [Platform Data Containers](data-ingestion-and-preparation/README.ipynb#platform-data-containers)):\n",
" ```sh\n",
" conda env -n myenv export > /User/virtual_env/myenv.yaml\n",
" ```\n",
" Using PIP:\n",
" ```sh\n",
" pip install <package> [<package> ...]\n",
" ```\n",
" <br>Alternatively, using Conda:\n",
" ```sh\n",
" conda install <package> [<package> ...]\n",
" ```\n",
" \n",
" <br>For example, the following command uses PIP to install the SciPy, pandas version 1.4.4, and TensorFlow version 2.9.3 packages for the \"myenv\" environment that you activated in the previous step:\n",
" ```sh\n",
" pip install scipy pandas==1.4.4 tensorflow==2.9.3\n",
" ```\n",
" <br>\n",
"\n",
"5. <a id=\"conda-virt-env-setup-step-export-env\"></a>Export your new virtual environment to an environment file in a platform data container by running the following command.\n",
" Replace `<container name>` with the name of a platform data container, `<directory path>` with an optional relative container-directories path, and `<environment name>` with the name of the environment that you created:\n",
" ```sh\n",
" conda env export -n <environment name> > /v3io/<container name>[/<directory path>]/<environment name>.yaml\n",
" ```\n",
" <br>It is recommended that you save the environment to a virtual-environments directory in your running-user home directory (**/v3io/users/&lt;running user&gt;**).\n",
" \n",
" For example, the following command creates a **/v3io/users/&lt;running user&gt;/virtual_env/myenv.yaml** file:\n",
" ```sh\n",
" conda env export -n myenv > /v3io/users/$V3IO_USERNAME/virtual_env/myenv.yaml\n",
" ```\n",
" \n",
" <br>To shorten this command, use the `/User` data mount to the running-user directory (see [Platform Data Containers](data-ingestion-and-preparation/README.ipynb#platform-data-containers)):\n",
" ```sh\n",
" conda env -n myenv export > /User/virtual_env/myenv.yaml\n",
" ```\n",
" <br>\n",
"\n",
"5. <a id=\"conda-virt-env-setup-step-refresh-ui\"></a>Refresh the JupyterLab UI to apply your changes.\n",
" After refreshing the UI, you should be able to see your new environment in the list of available kernels in Jupyter Notebook."
"6. <a id=\"conda-virt-env-setup-step-refresh-ui\"></a>Refresh the JupyterLab UI to apply your changes.<br>\n",
" After refreshing the UI, you should see your new environment in the list of available kernels in JupyterLab."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"<a id=\"creating-a-conda-environment-from-a-file\"></a>\n",
"### Creating a conda Virtual Environment from a File\n",
"### Creating a Conda Virtual Environment from a File\n",
"\n",
"If, for any reason, your conda environment variable is removed from Jupyter Notebook, you can easily deploy it again by using the YAML environment file that you exported in [Step 4](#conda-virt-env-setup-step-export-env) of the setup procedure:\n",
"If, for any reason, your Conda environment is removed from JupyterLab, you can easily deploy it again by using the YAML environment file that you exported in [Step 5](#conda-virt-env-setup-step-export-env) of the setup procedure:\n",
"\n",
"1. Open a new Jupyter terminal.\n",
"\n",
Expand All @@ -107,24 +150,50 @@
" conda env create --file /v3io/<container name>[/<directory path>]/<environment name>.yaml\n",
" ```\n",
"\n",
" For example, the following command loads a **users/&lt;running user&gt;/virtual_env/myenv.yaml** environment file; the command uses the `User` running-user directory data mount to the running-user directory in the \"users\" container:\n",
" <br>For example, the following command loads a /v3io/users/&lt;running user&gt;/virtual_env/myenv.yaml environment file.<br>\n",
" The command uses the `/User` running-user directory data mount to the running-user directory in the \"users\" container:\n",
" ```sh\n",
" conda env create --file /User/virtual_env/myenv.yaml\n",
" ```"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"<a id=\"cloning-an-existing-conda-environment\"></a>\n",
"### Cloning an existing Conda Virtual Environment\n",
"\n",
"To clone an existing Conda environment into a new one, follow these steps:\n",
"\n",
"1. Open a new Jupyter terminal.\n",
"\n",
"2. Create a new Conda enviroment by cloning an existing one.<br>\n",
" Replace `<environment name>` with the name of the new environmnent and `<source environment name>` with the environment to be cloned:\n",
" ```sh\n",
" conda create -n <environment name> --clone <source environment name>\n",
" ```\n",
" \n",
" <br>For example, the following command clones \"mlrun-base\" environment into \"mlrun-clone\":\n",
" ```sh\n",
" conda create -n mlrun-clone --clone mlrun-base\n",
" ``` \n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"<a id=\"setting-up-a-rapids-conda-environment\"></a>\n",
"## Setting Up a RAPIDS conda Environment with cuDF and cuML\n",
"## Setting Up a RAPIDS Conda Environment with cuDF and cuML\n",
"\n",
"To use the cuDF and cuML RAPIDS libraries, you need to create a RAPIDS conda environment.\n",
"To use the cuDF and cuML RAPIDS libraries, you need to create a RAPIDS Conda environment.\n",
"Use the following command to create a RAPIDS conda environment named `rapids`:\n",
"\n",
"```sh\n",
"conda create -n rapids -c rapidsai -c nvidia -c anaconda -c conda-forge -c defaults ipykernel rapids=0.17 python=3.7 cudatoolkit=11.0\n",
"conda create -n rapids -c rapidsai -c conda-forge -c nvidia rapids=22.12 cudatoolkit=11.7.0\n",
"```"
]
}
Expand All @@ -150,4 +219,4 @@
},
"nbformat": 4,
"nbformat_minor": 4
}
}

0 comments on commit def0cf7

Please sign in to comment.