Skip to content

Commit

Permalink
Here's a summary of the changes from the last pull request:
Browse files Browse the repository at this point in the history
- [🎉 NEW] We now support remote cluster creation on any Azure Cloud
  Environment, abstracting the requirements of different cloud
  environments and backend services. This means we now support
  deployments in sovereign clouds, such as the [US Government
  Cloud](https://learn.microsoft.com/en-us/azure/azure-government/documentation-government-welcome),
  the [German Cloud](https://learn.microsoft.com/en-us/azure/germany/),
  and the [China Cloud](https://learn.microsoft.com/en-us/azure/china/overview-operations),
  in addition to the Azure Public Cloud.
- [📈 IMPROVEMENT] We upgraded terraform provider to version 3.89.0 to
  support cluster setup through Azure Cloud Shell.
- [⚒️ FIX] We fixed an issue that would cause the local cluster setup to
  fail in case the cluster name had underscores. Related to issue #133.
- [📈 IMPROVEMENT] We revised the notebook dependencies and changed the
  default environment and package manager from Conda to
  [Micromamba](https://mamba.readthedocs.io/en/latest/installation/micromamba-installation.html).
  Environments should build considerably faster and present fewer
  dependency issues. Related to issues #141 and #144.
- [🎉 NEW] We added instructions on setting up the remote cluster using
  Azure Cloud Shell to the [Remote Cluster User
  Guide](https://microsoft.github.io/farmvibes-ai/docfiles/markdown/AKS.html).
- [⚒️ FIX] We fixed a bug that prevented the REST API documentation page
  to be rendered correctly. Related to issue #131.

----

Co-authored-by: Bruno Silva <brunosilva@microsoft.com>
Co-authored-by: Peter Hammond <v-phammond@microsoft.com>
Co-authored-by: Rafael Padilha <rpadilha@microsoft.com>
Co-authored-by: Renato Luiz de Freitas Cunha <renato.cunha@microsoft.com>
Co-authored-by: Rick Gutierrez <ricardgu@microsoft.com>
  • Loading branch information
6 people committed Feb 8, 2024
1 parent cf17eb1 commit f85c22a
Show file tree
Hide file tree
Showing 43 changed files with 504 additions and 229 deletions.
103 changes: 92 additions & 11 deletions docs/source/docfiles/markdown/AKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,16 +10,78 @@ cluster and its associated Azure cloud components.

## Requirements

The FarmVibes.AI remote management script works with the assumption that
whoever is executing it has at least a *Contributor* role in the Azure
subscription.
The FarmVibes.AI remote management script is
a [Python](https://python.org)-based utility that works with the assumption that whoever is
executing it has at least a *Contributor* role in the Azure subscription.
Please make sure you have Python installed before trying to setup a cluster.

The `az` command-line interface is a hard requirement for running the script.
Please follow the [Azure Command-Line Interface (CLI)
documentation](https://docs.microsoft.com/cli/azure/) for instructions on how
to install it. Make sure to install the version appropriate for your
architecture, otherwise the install process may fail.

If you have issues installing these utilities, please see the next section,
with instructions on how to build a FarmVibes.AI remote cluster from an Azure
Cloud Shell.

### Required Azure CPU quotas

In the default configuration, your subscription's CPU quota must support at
least 20 vCPUs in the region you choose. You will also need 12 `Standard DSv3
Family vCPUs` available. Please follow the [Azure quota increase
guide](https://learn.microsoft.com/en-us/azure/quotas/quickstart-increase-quota-portal).
The installer will fail if the quota is not available.

To summarize, the minimum numbers of CPUs required are:

* Total Regional vCPUs: increase to at least 20
* Standard DSv3 Family vCPUs: increase to at least 12
* Standard BS Family vCPUs: increase to at least 8

You might have to enable the `Microsoft.Compute` provider in your subscription.
For instructions on how to do so, please proceed to the [Azure
providers](#azure-providers) section.

### Azure Cloud Shell installation

For cases when users are unable to install the FarmVibes.AI requirements in
a local machine, they can use an Azure Cloud Shell to complete installation.

To do so, please visit [https://shell.azure.com/](https://shell.azure.com/) to
start a new Azure Cloud Shell.

When asked whether to start a "Bash" or "PowerShell" shell, select "Bash".

Then, do the following manual steps:

```bash
pip install --upgrade pip # to use the latest version of the package manager
# Do the following to install the `vibe_core` library and the `farmvibes-ai` command
pip install "git+https://github.com/microsoft/farmvibes-ai#egg =vibe_core&subdirectory=src/vibe_core"
az provider register --namespace Microsoft.Compute # to enable compute provider
```

After running the above commands, please manually set the processor quota in
the Azure region you want to use, if you haven't done so. (See the section
[Required Azure CPU quotas](#required-azure-cpu-quotas) for details.)

Then, run an `az account show` and make sure you are connected to the correct
Azure subscription. If not, `az account list` will list all your subscriptions,
and running an `az account set $SUBSCRIPTION_ID` will set the current
subscription, where `$SUBSCRIPTION_ID` is the id of the subscription you'd like
to use.

Depending on your permissions, you may or may not need to create a resource
group manually to install the FarmVibes.AI AKS cluster to. If needed, you can
create a new resource group with `az group create --name resource_group_name
--location location_name`, where `resource_group_name` is the name of the
resource group, and `location_name` is the Azure Region where you will install
the cluster.

Once these requirements are met, you can follow the instructions on how to use
[the farmvibes-ai script](#the-farmvibes-ai-script).

### Azure Providers

Since the FarmVibes.AI remote management script needs to provision new
Expand All @@ -37,14 +99,6 @@ writing of this document, the required providers are:
* `Microsoft.Compute`, for provisioning Virtual Machines and Virtual Machines
Scale Sets

### CPUs

In the default configuration, your subscription's CPU quota must support at
least 16 vCPUs in the region you choose. You will also need 12 `Standard DSv3
Family vCPUs` available. Please follow the [Azure quota increase
guide](https://learn.microsoft.com/en-us/azure/quotas/quickstart-increase-quota-portal).
The installer will fail if the quota is not available.

### Setting the subscription

Some Azure users will have access to multiple subscriptions. To list all
Expand All @@ -60,6 +114,28 @@ Then, you can set your default subscription with the command
az account set ${YOUR_DESIRED_SUBSCRIPTION_ID}
```

## Azure environments

The AKS environment supports all [Azure (National) Cloud
environments](https://learn.microsoft.com/en-us/graph/deployments). Users can
choose which cloud they want to interact with by using the `--environment`
argument. Valid values for this argument are:

* `public` (the default if not specified)
* `usgovernment` (for a US Government cloud)
* `german` (for the German sovereign cloud)
* `china` (for the China sovereign cloud)

## Overriding the cluster admin

The `farmvibes-ai` install script uses the `az` CLI to infer the credentials of
the administrator of the cluster (= the user currently-logged in). In some
situations, the user performing the install might not be the user that will
actually manage the cluster with `kubectl`.

In such cases, you can use the `--cluster-admin-name` to define who will have
access to the cluster.

## Azure Cloud Components

![Architecture diagram of the remote FarmVibes.AI configuration](./aks-terraform.svg)
Expand Down Expand Up @@ -193,6 +269,8 @@ instructions on how to install git.

4. After installing the `vibe_core` package, you can run the `farmvibes-ai` script.

### The farmvibes-ai script

The `farmvibes-ai` script manages the full lifecycle of the remote cluster,
including creation, deletion, and updates. The script is organized in
subcommands, each one responsible for a specific action. The subcommands, with
Expand All @@ -207,6 +285,9 @@ The script will create a new resource group with the specified name, and use it
If you ever need to destroy your cluster, all changes in your subscription will
be restricted to that resource group.

**WARNING**: If you decide to reuse an already existing resource group,
destroying the cluster will delete **ALL** resources in that resource group.

An example setup command is shown below:

```bash
Expand Down
3 changes: 2 additions & 1 deletion docs/source/docfiles/markdown/NOTEBOOK_LIST.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,6 @@ We organize available notebooks in the following topics:


</details>

<details>
<summary> Index Computation </summary>

Expand Down Expand Up @@ -160,6 +159,8 @@ We organize available notebooks in the following topics:

- [`Harvest and germination periods` 📓](https://github.com/microsoft/farmvibes-ai/blob/main/notebooks/harvest_period/ndvi_summary.ipynb)

- [`Investigating Amazon Rainforest deforestation with SpaceEye` 📓](https://github.com/microsoft/farmvibes-ai/blob/main/notebooks/sentinel/sentinel_spaceeye.ipynb)

- [`Irrigation classification` 📓](https://github.com/microsoft/farmvibes-ai/blob/main/notebooks/irrigation/irrigation_classification.ipynb)

- [`Land degradation` 📓](https://github.com/microsoft/farmvibes-ai/blob/main/notebooks/land_degradation/land_degradation.ipynb)
Expand Down
4 changes: 2 additions & 2 deletions docs/source/docfiles/markdown/TROUBLESHOOTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -220,7 +220,7 @@ This document compiles the most common issues encountered when installing and ru
<details>
<summary> Unable to import modules when running a notebook</summary>
Make sure you have installed and activated the conda environment provided with the notebook.
Make sure you have installed and activated the [micromamba environment](https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html) provided with the notebook.
</details>
Expand Down Expand Up @@ -250,7 +250,7 @@ This document compiles the most common issues encountered when installing and ru
`farmvibes-ai local add-onnx` command. If you are using a different storage location,
make sure to pass the `--storage-path` flag to the `add-onnx` command.

Before running the script, make sure you have a conda environment set up with the required
Before running the script, make sure you have a micromamba environment set up with the required
packages. You can use the environments defined by `env_cpu.yaml` or `env_gpu.yaml` files in the
`notebooks/segment_anything` directory.

Expand Down
1 change: 0 additions & 1 deletion docs/source/docfiles/markdown/WORKFLOW_LIST.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,6 @@ Below is a list of all available workflows within the FarmVibes.AI platform. For
- [`heatmap/classification` 📄](workflow_yaml/data_processing/heatmap/classification.md): Utilizes input Sentinel-2 satellite imagery & the sensor samples as labeled data that contain nutrient information (Nitrogen, Carbon, pH, Phosphorus) to train a model using Random Forest classifier. The inference operation predicts nutrients in soil for the chosen farm boundary.



- [`index/index` 📄](workflow_yaml/data_processing/index/index.md): Computes an index from the bands of an input raster.

- [`linear_trend/chunked_linear_trend` 📄](workflow_yaml/data_processing/linear_trend/chunked_linear_trend.md): Computes the pixel-wise linear trend of a list of rasters (e.g. NDVI).
Expand Down
10 changes: 5 additions & 5 deletions notebooks/README.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
# FarmVibes.AI Example Notebooks

The notebooks presented here serve as examples on the how to get started, and some of the capabilities of FarmVibes.AI.
We recommend installing the required packages to run the notebooks within a conda environment.
We recommend installing the required packages to run the notebooks within a micromamba environment.

## Installing and using conda
## Installing and using micromamba

Please see the instalation instructions in the [Anaconda website](https://docs.conda.io/en/latest/miniconda.html).
Also check [instructions for getting started with conda](https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html) and the [conda user guide](https://docs.conda.io/projects/conda/en/latest/user-guide/index.html) .
Please see the instalation instructions in the [micromamba website](https://mamba.readthedocs.io/en/latest/).
Also check the [micromamba user guide](https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html) and the [troubleshoooting page](https://mamba.readthedocs.io/en/latest/user_guide/troubleshooting.html).

## Installing the environment

A set of common packages that should enable you to several notebooks is present in the environment described in `env.yaml`. To create the environment, run

```bash
conda env create -f env.yaml
micromamba env create -f env.yaml
```

This will create an environment named `farmvibes-ai`.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,19 @@
"![image-4.png](attachment:image-4.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Micromamba environment setup\n",
"To install the required packages, see [this README file](../README.md). You can activate the environment with the following command:\n",
"\n",
"\n",
"```bash\n",
"$ micromamba activate farmvibes-ai\n",
"```"
]
},
{
"attachments": {},
"cell_type": "markdown",
Expand Down
13 changes: 13 additions & 0 deletions notebooks/admag/azure_data_manager_for_agriculture_example.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,19 @@
"In this notebook, we will explain how to connect FarmVibes.AI with [Microsoft Azure Data Manager for Agriculture](https://aka.ms/farmvibesDMA), and provide an example of how to leverage the FarmVibes.AI workflows using ADMAg inputs. We will demonstrate how to compose the ADMAg and NDVI summary workflows into a single custom workflow, and check the results for the user's agriculture field."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Micromamba environment setup\n",
"To install the required packages, see [this README file](../README.md). You can activate the environment with the following command:\n",
"\n",
"\n",
"```bash\n",
"$ micromamba activate farmvibes-ai\n",
"```"
]
},
{
"attachments": {},
"cell_type": "markdown",
Expand Down
4 changes: 2 additions & 2 deletions notebooks/carbon/whatif.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@
"\n",
"This notebook uses the [COMET-Farm API](https://gitlab.com/comet-api/api-docs/-/tree/master/) to derive soil carbon sequestration information for agricultural fields.\n",
"\n",
"### Conda environment setup\n",
"### Micromamba environment setup\n",
"\n",
"To install the required packages, please refer to our [documentation](https://github.com/microsoft/farmvibes-ai).\n",
"To install the required packages, please refer to our [documentation](https://github.com/microsoft/farmvibes-ai/tree/main/notebooks).\n",
"\n",
"### Required Input\n",
"\n",
Expand Down
28 changes: 17 additions & 11 deletions notebooks/crop_cycles/crop_cycles.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,12 @@
"\n",
"This notebook illustrates how to use the `chunk_onnx` workflow to run time-series analysis over raster images (in parallel). Specifically, here we count the number of crop cycles in a year a certain region has. However, one can define other time analysis as long as that can be defined as an Onnx model. \n",
"\n",
"### Conda environment setup\n",
"Before running this notebook, let's build a conda environment. If you do not have conda installed, please follow the instructions from [Conda User Guide](https://docs.conda.io/projects/conda/en/latest/user-guide/index.html). \n",
"### Micromamba environment setup\n",
"Before running this notebook, let's build a micromamba environment. If you do not have micromamba installed, please follow the instructions from the [micromamba installation page](https://mamba.readthedocs.io/en/latest/installation/micromamba-installation.html).\n",
"\n",
"```\n",
"$ conda env create -f ./env.yaml\n",
"$ conda activate crop_cycles\n",
"```bash\n",
"$ micromamba env create -f ./env.yaml\n",
"$ micromamba activate crop_cycles\n",
"```"
]
},
Expand Down Expand Up @@ -347,6 +347,8 @@
}
],
"metadata": {
"description": "Run time-series analysis over NDVI data to detect the number of crop cycles in a year for a certain region.",
"disk_space": "",
"kernelspec": {
"display_name": "dev-vibes",
"language": "python",
Expand All @@ -364,16 +366,20 @@
"pygments_lexer": "ipython3",
"version": "3.8.15"
},
"name": "Crop cycle detection",
"running_time": "",
"tags": [
"ONNX Integration",
"Remote Sensing",
"Working with Custom Workflows",
"SpaceEye",
"Index Computation"
],
"vscode": {
"interpreter": {
"hash": "eb30320db5fa0962177f317f4aa59da24b9a1536f3ecf9dd746c9d96cf84cbcc"
}
},
"name": "Crop cycle detection",
"description": "Run time-series analysis over NDVI data to detect the number of crop cycles in a year for a certain region.",
"tags": ["ONNX Integration", "Remote Sensing", "Working with Custom Workflows", "SpaceEye", "Index Computation"],
"disk_space": "",
"running_time": ""
}
},
"nbformat": 4,
"nbformat_minor": 2
Expand Down
27 changes: 16 additions & 11 deletions notebooks/crop_segmentation/01_dataset_generation.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,12 @@
"As provided, this notebook retrieves and preprocesses a region of ~5,000 km² over a 1-year period. **We recommend having at least 500 GB of disk space available. The workflow may take multiple days to run, depending on the number of workers and your VM spec.**\n",
"\n",
"\n",
"### Conda environment setup\n",
"Before running this notebook, let's build a conda environment. If you do not have conda installed, please follow the instructions from [Conda User Guide](https://docs.conda.io/projects/conda/en/latest/user-guide/index.html). \n",
"### Micromamba environment setup\n",
"Before running this notebook, let's build a micromamba environment. If you do not have micromamba installed, please follow the instructions from the [micromamba installation guide](https://mamba.readthedocs.io/en/latest/installation/micromamba-installation.html). \n",
"\n",
"```\n",
"$ conda env create -f ./crop_env.yaml\n",
"$ conda activate crop-seg\n",
"```bash\n",
"$ micromamba env create -f ./crop_env.yaml\n",
"$ micromamba activate crop-seg\n",
"```\n",
"\n",
"### Notebook outline\n",
Expand Down Expand Up @@ -432,6 +432,8 @@
}
],
"metadata": {
"description": "Generate a dataset based on NDVI and CDL rasters to train a crop land segmentation model.",
"disk_space": "",
"kernelspec": {
"display_name": "Python 3.8.13 ('crop-seg')",
"language": "python",
Expand All @@ -449,16 +451,19 @@
"pygments_lexer": "ipython3",
"version": "3.8.13"
},
"name": "Crop land segmentation (1/4) - dataset generation",
"running_time": "",
"tags": [
"Crop Land Segmentation",
"Model Training",
"Working with Custom Workflows",
"Remote Sensing"
],
"vscode": {
"interpreter": {
"hash": "c4ed1382910929959896ae46210007cc63f041dd0c6059830101875a13ee5841"
}
},
"name": "Crop land segmentation (1/4) - dataset generation",
"description": "Generate a dataset based on NDVI and CDL rasters to train a crop land segmentation model.",
"tags": ["Crop Land Segmentation", "Model Training", "Working with Custom Workflows", "Remote Sensing"],
"disk_space": "",
"running_time": ""
}
},
"nbformat": 4,
"nbformat_minor": 2
Expand Down
10 changes: 5 additions & 5 deletions notebooks/crop_segmentation/02_visualize_dataset.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,12 @@
"We run a workflow that exposes the intermediate outputs of the dataset generation workflow, allowing us to visualize Sentinel-2 data, NDVI rasters and [Crop Data Layer](https://data.nal.usda.gov/dataset/cropscape-cropland-data-layer#:~:text=The%20Cropland%20Data%20Layer%20%28CDL%29%2C%20hosted%20on%20CropScape%2C,as%20well%20as%20boundary%2C%20water%20and%20road%20layers.) (CDL) maps provided by the FarmVibes.AI platform.\n",
"\n",
"\n",
"### Conda environment setup\n",
"Before running this notebook, let's build a conda environment. If you do not have conda installed, please follow the instructions from [Conda User Guide](https://docs.conda.io/projects/conda/en/latest/user-guide/index.html). \n",
"### Micromamba environment setup\n",
"Before running this notebook, let's build a micromamba environment. If you do not have micromamba installed, please follow the instructions from the [micromamba installation guide](https://mamba.readthedocs.io/en/latest/installation/micromamba-installation.html).\n",
"\n",
"```\n",
"$ conda env create -f ./crop_env.yaml\n",
"$ conda activate crop-seg\n",
"```bash\n",
"$ micromamba env create -f ./crop_env.yaml\n",
"$ micromamba activate crop-seg\n",
"```\n",
"\n",
"------------------------\n",
Expand Down
Loading

0 comments on commit f85c22a

Please sign in to comment.