Support for multiple Azure Cloud Environments and other improvements (#…

…146)

- [🎉 NEW] We now support remote cluster creation on any Azure Cloud
Environment, abstracting the requirements of different cloud
environments and backend services. This means we now support deployments
in sovereign clouds, such as the [US Government
Cloud](https://learn.microsoft.com/en-us/azure/azure-government/documentation-government-welcome),
the [German Cloud](https://learn.microsoft.com/en-us/azure/germany/),
and the [China
Cloud](https://learn.microsoft.com/en-us/azure/china/overview-operations),
in addition to the Azure Public Cloud.
- [📈 IMPROVEMENT] We upgraded terraform provider to version 3.89.0 to
support cluster setup through Azure Cloud Shell.
- [⚒️ FIX] We fixed an issue that would cause the local cluster setup to
fail in case the cluster name had underscores. Related to issue #133.
- [📈 IMPROVEMENT] We revised the notebook dependencies and changed the
default environment and package manager from Conda to
[Micromamba](https://mamba.readthedocs.io/en/latest/installation/micromamba-installation.html).
Environments should build considerably faster and present fewer
dependency issues. Related to issues #141 and #144.
- [🎉 NEW] We added instructions on setting up the remote cluster using
Azure Cloud Shell to the [Remote Cluster User
Guide](https://microsoft.github.io/farmvibes-ai/docfiles/markdown/AKS.html).
- [⚒️ FIX] We fixed a bug that prevented the REST API documentation page
to be rendered correctly. Related to issue #131.

----

Co-authored-by: FarmVibes Release Pipeline <farmvibesaicd@microsoft.com>
Co-authored-by: Bruno Silva <brunosilva@microsoft.com>
Co-authored-by: Peter Hammond <v-phammond@microsoft.com>
Co-authored-by: Rafael Padilha <rpadilha@microsoft.com>
Co-authored-by: Rick Gutierrez <ricardgu@microsoft.com>

docs/source/docfiles/markdown/AKS.md

@@ -10,16 +10,78 @@ cluster and its associated Azure cloud components.
 
 ## Requirements
 
-The FarmVibes.AI remote management script works with the assumption that
-whoever is executing it has at least a *Contributor* role in the Azure
-subscription.
+The FarmVibes.AI remote management script is
+a [Python](https://python.org)-based utility that works with the assumption that whoever is
+executing it has at least a *Contributor* role in the Azure subscription.
+Please make sure you have Python installed before trying to setup a cluster.
 
 The `az` command-line interface is a hard requirement for running the script.
 Please follow the [Azure Command-Line Interface (CLI)
 documentation](https://docs.microsoft.com/cli/azure/) for instructions on how
 to install it. Make sure to install the version appropriate for your
 architecture, otherwise the install process may fail.
 
+If you have issues installing these utilities, please see the next section,
+with instructions on how to build a FarmVibes.AI remote cluster from an Azure
+Cloud Shell.
+
+### Required Azure CPU quotas
+
+In the default configuration, your subscription's CPU quota must support at
+least 20 vCPUs in the region you choose. You will also need 12 `Standard DSv3
+Family vCPUs` available. Please follow the [Azure quota increase
+guide](https://learn.microsoft.com/en-us/azure/quotas/quickstart-increase-quota-portal).
+The installer will fail if the quota is not available.
+
+To summarize, the minimum numbers of CPUs required are:
+
+ * Total Regional vCPUs: increase to at least 20
+ * Standard DSv3 Family vCPUs: increase to at least 12
+ * Standard BS Family vCPUs: increase to at least 8
+
+You might have to enable the `Microsoft.Compute` provider in your subscription.
+For instructions on how to do so, please proceed to the [Azure
+providers](#azure-providers) section.
+
+### Azure Cloud Shell installation
+
+For cases when users are unable to install the FarmVibes.AI requirements in
+a local machine, they can use an Azure Cloud Shell to complete installation.
+
+To do so, please visit [https://shell.azure.com/](https://shell.azure.com/) to
+start a new Azure Cloud Shell.
+
+When asked whether to start a "Bash" or "PowerShell" shell, select "Bash".
+
+Then, do the following manual steps:
+
+```bash
+pip install --upgrade pip  # to use the latest version of the package manager
+# Do the following to install the `vibe_core` library and the `farmvibes-ai` command
+pip install "git+https://github.com/microsoft/farmvibes-ai#egg =vibe_core&subdirectory=src/vibe_core"
+az provider register --namespace Microsoft.Compute  # to enable compute provider
+```
+
+After running the above commands, please manually set the processor quota in
+the Azure region you want to use, if you haven't done so. (See the section
+[Required Azure CPU quotas](#required-azure-cpu-quotas) for details.)
+
+Then, run an `az account show` and make sure you are connected to the correct
+Azure subscription. If not, `az account list` will list all your subscriptions,
+and running an `az account set $SUBSCRIPTION_ID` will set the current
+subscription, where `$SUBSCRIPTION_ID` is the id of the subscription you'd like
+to use.
+
+Depending on your permissions, you may or may not need to create a resource
+group manually to install the FarmVibes.AI AKS cluster to. If needed, you can
+create a new resource group with `az group create --name resource_group_name
+--location location_name`, where `resource_group_name` is the name of the
+resource group, and `location_name` is the Azure Region where you will install
+the cluster.
+
+Once these requirements are met, you can follow the instructions on how to use
+[the farmvibes-ai script](#the-farmvibes-ai-script).
+
 ### Azure Providers
 
 Since the FarmVibes.AI remote management script needs to provision new
@@ -37,14 +99,6 @@ writing of this document, the required providers are:
  * `Microsoft.Compute`, for provisioning Virtual Machines and Virtual Machines
    Scale Sets
 
-### CPUs
-
-In the default configuration, your subscription's CPU quota must support at
-least 16 vCPUs in the region you choose. You will also need 12 `Standard DSv3
-Family vCPUs` available. Please follow the [Azure quota increase
-guide](https://learn.microsoft.com/en-us/azure/quotas/quickstart-increase-quota-portal).
-The installer will fail if the quota is not available.
-
 ### Setting the subscription
 
 Some Azure users will have access to multiple subscriptions. To list all
@@ -60,6 +114,28 @@ Then, you can set your default subscription with the command
 az account set ${YOUR_DESIRED_SUBSCRIPTION_ID}
 ```
 
+## Azure environments
+
+The AKS environment supports all [Azure (National) Cloud
+environments](https://learn.microsoft.com/en-us/graph/deployments). Users can
+choose which cloud they want to interact with by using the `--environment`
+argument. Valid values for this argument are:
+
+ * `public` (the default if not specified)
+ * `usgovernment` (for a US Government cloud)
+ * `german` (for the German sovereign cloud)
+ * `china` (for the China sovereign cloud)
+
+## Overriding the cluster admin
+
+The `farmvibes-ai` install script uses the `az` CLI to infer the credentials of
+the administrator of the cluster (= the user currently-logged in). In some
+situations, the user performing the install might not be the user that will
+actually manage the cluster with `kubectl`.
+
+In such cases, you can use the `--cluster-admin-name` to define who will have
+access to the cluster.
+
 ## Azure Cloud Components
 
 ![Architecture diagram of the remote FarmVibes.AI configuration](./aks-terraform.svg)
@@ -193,6 +269,8 @@ instructions on how to install git.
 
 4. After installing the `vibe_core` package, you can run the `farmvibes-ai` script.
 
+### The farmvibes-ai script
+
 The `farmvibes-ai` script manages the full lifecycle of the remote cluster,
 including creation, deletion, and updates. The script is organized in
 subcommands, each one responsible for a specific action. The subcommands, with
@@ -207,6 +285,9 @@ The script will create a new resource group with the specified name, and use it
 If you ever need to destroy your cluster, all changes in your subscription will
 be restricted to that resource group.
 
+**WARNING**: If you decide to reuse an already existing resource group,
+destroying the cluster will delete **ALL** resources in that resource group.
+
 An example setup command is shown below:
 
 ```bash

docs/source/docfiles/markdown/NOTEBOOK_LIST.md

@@ -62,7 +62,6 @@ We organize available notebooks in the following topics:
 
 
 </details>
-
 <details>
 <summary> Index Computation </summary>
 
@@ -160,6 +159,8 @@ We organize available notebooks in the following topics:
 
 - [`Harvest and germination periods` 📓](https://github.com/microsoft/farmvibes-ai/blob/main/notebooks/harvest_period/ndvi_summary.ipynb)
 
+- [`Investigating Amazon Rainforest deforestation with SpaceEye` 📓](https://github.com/microsoft/farmvibes-ai/blob/main/notebooks/sentinel/sentinel_spaceeye.ipynb)
+
 - [`Irrigation classification` 📓](https://github.com/microsoft/farmvibes-ai/blob/main/notebooks/irrigation/irrigation_classification.ipynb)
 
 - [`Land degradation` 📓](https://github.com/microsoft/farmvibes-ai/blob/main/notebooks/land_degradation/land_degradation.ipynb)

docs/source/docfiles/markdown/TROUBLESHOOTING.md

@@ -220,7 +220,7 @@ This document compiles the most common issues encountered when installing and ru
   <details>
   <summary> Unable to import modules when running a notebook</summary>
 
-  Make sure you have installed and activated the conda environment provided with the notebook.
+  Make sure you have installed and activated the [micromamba environment](https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html) provided with the notebook.
 
   </details>
 
@@ -250,7 +250,7 @@ This document compiles the most common issues encountered when installing and ru
   `farmvibes-ai local add-onnx` command. If you are using a different storage location,
   make sure to pass the `--storage-path` flag to the `add-onnx` command.
 
-  Before running the script, make sure you have a conda environment set up with the required
+  Before running the script, make sure you have a micromamba environment set up with the required
   packages. You can use the environments defined by `env_cpu.yaml` or `env_gpu.yaml` files in the
   `notebooks/segment_anything` directory.
 

notebooks/README.md

@@ -1,19 +1,19 @@
 # FarmVibes.AI Example Notebooks
 
 The notebooks presented here serve as examples on the how to get started, and some of the capabilities of FarmVibes.AI.
-We recommend installing the required packages to run the notebooks within a conda environment.
+We recommend installing the required packages to run the notebooks within a micromamba environment.
 
-## Installing and using conda
+## Installing and using micromamba
 
-Please see the instalation instructions in the [Anaconda website](https://docs.conda.io/en/latest/miniconda.html).
-Also check [instructions for getting started with conda](https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html) and the [conda user guide](https://docs.conda.io/projects/conda/en/latest/user-guide/index.html) .
+Please see the instalation instructions in the [micromamba website](https://mamba.readthedocs.io/en/latest/).
+Also check the [micromamba user guide](https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html) and the [troubleshoooting page](https://mamba.readthedocs.io/en/latest/user_guide/troubleshooting.html).
 
 ## Installing the environment
 
 A set of common packages that should enable you to several notebooks is present in the environment described in `env.yaml`. To create the environment, run
 
 ```bash
-conda env create -f env.yaml
+micromamba env create -f env.yaml
 ```
 
 This will create an environment named `farmvibes-ai`.

notebooks/admag/azure_data_manager_for_agriculture_and_comet_farm_api_example.ipynb

@@ -29,6 +29,19 @@
     "![image-4.png](attachment:image-4.png)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Micromamba environment setup\n",
+    "To install the required packages, see [this README file](../README.md). You can activate the environment with the following command:\n",
+    "\n",
+    "\n",
+    "```bash\n",
+    "$ micromamba activate farmvibes-ai\n",
+    "```"
+   ]
+  },
   {
    "attachments": {},
    "cell_type": "markdown",

notebooks/admag/azure_data_manager_for_agriculture_example.ipynb

@@ -11,6 +11,19 @@
     "In this notebook, we will explain how to connect FarmVibes.AI with [Microsoft Azure Data Manager for Agriculture](https://aka.ms/farmvibesDMA), and provide an example of how to leverage the FarmVibes.AI workflows using ADMAg inputs. We will demonstrate how to compose the ADMAg and NDVI summary workflows into a single custom workflow, and check the results for the user's agriculture field."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Micromamba environment setup\n",
+    "To install the required packages, see [this README file](../README.md). You can activate the environment with the following command:\n",
+    "\n",
+    "\n",
+    "```bash\n",
+    "$ micromamba activate farmvibes-ai\n",
+    "```"
+   ]
+  },
   {
    "attachments": {},
    "cell_type": "markdown",

notebooks/carbon/whatif.ipynb

@@ -8,9 +8,9 @@
     "\n",
     "This notebook uses the [COMET-Farm API](https://gitlab.com/comet-api/api-docs/-/tree/master/) to derive soil carbon sequestration information for agricultural fields.\n",
     "\n",
-    "### Conda environment setup\n",
+    "### Micromamba environment setup\n",
     "\n",
-    "To install the required packages, please refer to our [documentation](https://github.com/microsoft/farmvibes-ai).\n",
+    "To install the required packages, please refer to our [documentation](https://github.com/microsoft/farmvibes-ai/tree/main/notebooks).\n",
     "\n",
     "### Required Input\n",
     "\n",

notebooks/crop_cycles/crop_cycles.ipynb

@@ -8,12 +8,12 @@
     "\n",
     "This notebook illustrates how to use the `chunk_onnx` workflow to run time-series analysis over raster images (in parallel). Specifically, here we count the number of crop cycles in a year a certain region has. However, one can define other time analysis as long as that can be defined as an Onnx model. \n",
     "\n",
-    "### Conda environment setup\n",
-    "Before running this notebook, let's build a conda environment. If you do not have conda installed, please follow the instructions from [Conda User Guide](https://docs.conda.io/projects/conda/en/latest/user-guide/index.html). \n",
+    "### Micromamba environment setup\n",
+    "Before running this notebook, let's build a micromamba environment. If you do not have micromamba installed, please follow the instructions from the [micromamba installation page](https://mamba.readthedocs.io/en/latest/installation/micromamba-installation.html).\n",
     "\n",
-    "```\n",
-    "$ conda env create -f ./env.yaml\n",
-    "$ conda activate crop_cycles\n",
+    "```bash\n",
+    "$ micromamba env create -f ./env.yaml\n",
+    "$ micromamba activate crop_cycles\n",
     "```"
    ]
   },
@@ -347,6 +347,8 @@
   }
  ],
  "metadata": {
+  "description": "Run time-series analysis over NDVI data to detect the number of crop cycles in a year for a certain region.",
+  "disk_space": "",
   "kernelspec": {
    "display_name": "dev-vibes",
    "language": "python",
@@ -364,16 +366,20 @@
    "pygments_lexer": "ipython3",
    "version": "3.8.15"
   },
+  "name": "Crop cycle detection",
+  "running_time": "",
+  "tags": [
+   "ONNX Integration",
+   "Remote Sensing",
+   "Working with Custom Workflows",
+   "SpaceEye",
+   "Index Computation"
+  ],
   "vscode": {
    "interpreter": {
     "hash": "eb30320db5fa0962177f317f4aa59da24b9a1536f3ecf9dd746c9d96cf84cbcc"
    }
-  },
-  "name": "Crop cycle detection",
-  "description": "Run time-series analysis over NDVI data to detect the number of crop cycles in a year for a certain region.",
-  "tags": ["ONNX Integration", "Remote Sensing", "Working with Custom Workflows", "SpaceEye", "Index Computation"],
-  "disk_space": "",
-  "running_time": ""
+  }
  },
  "nbformat": 4,
  "nbformat_minor": 2

notebooks/crop_segmentation/01_dataset_generation.ipynb

@@ -11,12 +11,12 @@
     "As provided, this notebook retrieves and preprocesses a region of ~5,000 km² over a 1-year period. **We recommend having at least 500 GB of disk space available. The workflow may take multiple days to run, depending on the number of workers and your VM spec.**\n",
     "\n",
     "\n",
-    "### Conda environment setup\n",
-    "Before running this notebook, let's build a conda environment. If you do not have conda installed, please follow the instructions from [Conda User Guide](https://docs.conda.io/projects/conda/en/latest/user-guide/index.html). \n",
+    "### Micromamba environment setup\n",
+    "Before running this notebook, let's build a micromamba environment. If you do not have micromamba installed, please follow the instructions from the [micromamba installation guide](https://mamba.readthedocs.io/en/latest/installation/micromamba-installation.html). \n",
     "\n",
-    "```\n",
-    "$ conda env create -f ./crop_env.yaml\n",
-    "$ conda activate crop-seg\n",
+    "```bash\n",
+    "$ micromamba env create -f ./crop_env.yaml\n",
+    "$ micromamba activate crop-seg\n",
     "```\n",
     "\n",
     "### Notebook outline\n",
@@ -432,6 +432,8 @@
   }
  ],
  "metadata": {
+  "description": "Generate a dataset based on NDVI and CDL rasters to train a crop land segmentation model.",
+  "disk_space": "",
   "kernelspec": {
    "display_name": "Python 3.8.13 ('crop-seg')",
    "language": "python",
@@ -449,16 +451,19 @@
    "pygments_lexer": "ipython3",
    "version": "3.8.13"
   },
+  "name": "Crop land segmentation (1/4) - dataset generation",
+  "running_time": "",
+  "tags": [
+   "Crop Land Segmentation",
+   "Model Training",
+   "Working with Custom Workflows",
+   "Remote Sensing"
+  ],
   "vscode": {
    "interpreter": {
     "hash": "c4ed1382910929959896ae46210007cc63f041dd0c6059830101875a13ee5841"
    }
-  },
-  "name": "Crop land segmentation (1/4) - dataset generation",
-  "description": "Generate a dataset based on NDVI and CDL rasters to train a crop land segmentation model.",
-  "tags": ["Crop Land Segmentation", "Model Training", "Working with Custom Workflows", "Remote Sensing"],
-  "disk_space": "",
-  "running_time": ""
+  }
  },
  "nbformat": 4,
  "nbformat_minor": 2

notebooks/crop_segmentation/02_visualize_dataset.ipynb

@@ -11,12 +11,12 @@
     "We run a workflow that exposes the intermediate outputs of the dataset generation workflow, allowing us to visualize Sentinel-2 data, NDVI rasters and [Crop Data Layer](https://data.nal.usda.gov/dataset/cropscape-cropland-data-layer#:~:text=The%20Cropland%20Data%20Layer%20%28CDL%29%2C%20hosted%20on%20CropScape%2C,as%20well%20as%20boundary%2C%20water%20and%20road%20layers.) (CDL) maps provided by the FarmVibes.AI platform.\n",
     "\n",
     "\n",
-    "### Conda environment setup\n",
-    "Before running this notebook, let's build a conda environment. If you do not have conda installed, please follow the instructions from [Conda User Guide](https://docs.conda.io/projects/conda/en/latest/user-guide/index.html). \n",
+    "### Micromamba environment setup\n",
+    "Before running this notebook, let's build a micromamba environment. If you do not have micromamba installed, please follow the instructions from the [micromamba installation guide](https://mamba.readthedocs.io/en/latest/installation/micromamba-installation.html).\n",
     "\n",
-    "```\n",
-    "$ conda env create -f ./crop_env.yaml\n",
-    "$ conda activate crop-seg\n",
+    "```bash\n",
+    "$ micromamba env create -f ./crop_env.yaml\n",
+    "$ micromamba activate crop-seg\n",
     "```\n",
     "\n",
     "------------------------\n",

notebooks/crop_segmentation/03_aml_training.ipynb

@@ -9,12 +9,12 @@
     "This notebook demonstrates how to train a neural network with Azure Machine Learning (AML) to segment crops on NDVI timeseries and [Crop Data Layer](https://data.nal.usda.gov/dataset/cropscape-cropland-data-layer#:~:text=The%20Cropland%20Data%20Layer%20%28CDL%29%2C%20hosted%20on%20CropScape%2C,as%20well%20as%20boundary%2C%20water%20and%20road%20layers.) (CDL) maps provided by FarmVibes.AI platform.\n",
     "\n",
     "\n",
-    "### Conda environment setup\n",
-    "Before running this notebook, let's build a conda environment. If you do not have conda installed, please follow the instructions from [Conda User Guide](https://docs.conda.io/projects/conda/en/latest/user-guide/index.html). \n",
+    "### Micromamba environment setup\n",
+    "Before running this notebook, let's build a micromamba environment. If you do not have micromamba installed, please follow the instructions from the [micromamba installation guide](https://mamba.readthedocs.io/en/latest/installation/micromamba-installation.html).\n",
     "\n",
-    "```\n",
-    "$ conda env create -f ./crop_env.yaml\n",
-    "$ conda activate crop-seg\n",
+    "```bash\n",
+    "$ micromamba env create -f ./crop_env.yaml\n",
+    "$ micromamba activate crop-seg\n",
     "```\n",
     "\n",
     "-----------"