Support to new ADMA API, SpaceEye improvements, Forest Extent workflows, and more!

[rafaspadilha]

Hello, FarmVibes.AI users! 👋

We are happy to announce a new release that brings exciting and impactful improvements.

We are releasing new workflows to download and process forest extent data, as well as important enhancements to the SpaceEye model. These are followed by new notebooks and a few quality-of-life improvements to our python client. On the backend, we've updated to Python 3.11, as well as improved the retry policy of our workers to reduce overhead in long queues.

There are two breaking changes that need your attention:

• We added support for the new Azure Data Manager for Agriculture (ADMA) API version 2023-11-01-preview, which led to updates in our data types, workflows, and notebooks to comply with the new API.
• Planetary Computer API key is now required for all workflows that download or use Sentinel-1 products.

Please, refer to the summarized release notes below for more details.

Now, to update your existing clusters, if you cloned the repo, do a

git pull

to update your local copy. Then, you will have to update (or install) the vibe_core library, with

pip install ./src/vibe_core

Finally, you can upgrade your cluster by running

farmvibes-ai local update

Thank you for using FarmVibes.AI!

---

Here's a summary of the changes from the last pull request:

Support to new ADMA API version:

[💔 BREAKING CHANGES] We added support to the new Azure Data Manager for Agriculture (ADMA) API version 2023-11-01-preview. To comply with the new API, some of our data types, workflows, and notebooks were updated. A summary of the changes are:

• The farmer_id property has now changed to party_id in ADMAgSeasonalFieldInput object (which is passed as input for our workflows) and other ADMA data types. Also, there is no more the boundary_id field as boundary resources are no more single objects in ADMA. The boundary information is part of the ADMA entities (seasonal field, field etc.).
• All operations (seasonal field, field, prescriptions etc.) now have their own geometry.
• Planting information in Seasonal Field was moved to a new operation "Planting Data".
• The affected workflows are: data_ingestion/admag/admag_seasonal_field, data_ingestion/admag/prescriptions, and farm_ai/agriculture/heatmap_using_classification_admag.
• Likewise, the following notebooks have been updated: notebooks/admag and notebooks/heatmaps/nutrients_using_classification_admag.

Workflows:

• [🎉 NEW] We added new workflows to download and process forest extent data. These workflows include: (i) the Advanced Land Observing Satellite (ALOS) forest/non-forest map, (ii) the Global Land Analysis (GLAD) forest map, and (iii) the Global Forest Change (Hansen) datasets. The processing workflow detects forest changes using statistical tests (specifically, the Cochran-Armitage test) to determine whether the change represents a trend.
• [📈 IMPROVEMENT] We are releasing an updated version of the SpaceEye workflow that leverages Sentinel-1 RTC products instead of GRD products with SNAP preprocessing. These changes lead to a significant speedup and a decrease in disk space consumption. Faster, lighter, and cloudless-er images to y'all! Due to this new version of the workflow, cached results are invalid and will be recomputed on new runs.
• [📈 IMPROVEMENT] We extended the data_ingestion/weather/download_herbie workflow to allow downloading both analysis (zero lead time) up to a point in time and then forecast with progressively increasing lead times after that.
• [⚒️ FIX] We added a script to convert Segment Anything Model (SAM) weights to ONNX format and import it into the local FarmVibes.AI cluster to be used in the SAM workflows and notebooks. For instructions on how to use the script, refer to the "Adding SAM's ONNX models to the cluster" entry in our Troubleshooting page. Related to issues Error running basemap_segmentation workflow - sam_inference failed #145 and Unable to export SAM models to cluster due to missing file scripts/export_sam_models.py #153.
• [💔 BREAKING CHANGES] Planetary Computer API key is required for all workflows that download or use Sentinel-1 products (e.g., data_ingestion/sentinel1/preprocess_s1, data_ingestion/spaceeye/spaceeye_preprocess, and data_ingestion/spaceeye/spaceeye). The documentation with the parameter description will be updated in a future release.

Backend:

• [📈 IMPROVEMENT] We updated our base images to use Python 3.11. Additionally, our DAPR logs are now in JSON format.
• [📈 IMPROVEMENT] We updated ngrok package to version 7.1.2, which will solve the issues with the What-if scenario evaluation for carbon sequestration notebook and farm_ai/carbon_local/carbon_whatif workflow. Related to issue What-if scenario evaluation for carbon sequestration notebook #151.
• [📈 IMPROVEMENT] We improved our worker retry policy and are now using exponential backoff for retries. This means workers will be less idle on long queues, avoiding cases where workers would wait for one minute before starting a queued task.
• [⚒️ FIX] We fixed an issue with the add-secret subcommand not adding secrets to the KeyVault during remote (AKS) cluster setup. Related to issue remote add-secret doesn’t work correctly #147.

Notebooks:

• [🎉 NEW] We added a few notebooks showcasing the new forest workflows, such as downloading ALOS and GLAD forest maps, and evaluating forest changes over time with the Hansen maps. We also provide a notebook highlighting how to detect increase/decrease trends in forest pixel levels over an user-input geometry and time range.

Documentation:

• [🎉 NEW] We added a new REST API documentation page describing the endpoints, parameters, and responses for our API.

Client:

• [📈 IMPROVEMENT & ⚒️ FIX] We made a few quality-of-life improvements and bug fixes to our python client. Users can now monitor the last n runs with client.monitor(n), which will print a summarized table with their progress, as well as retrieve the last n VibeWorkflowRun objects with client.get_last_runs(n) method. We also added minor bug fixes to the monitor and document_workflow methods, which were breaking for deleted runs and workflows with None/empty value parameters, respectively.

---

Co-authored-by: Bruno Silva brunosilva@microsoft.com
Co-authored-by: Eduardo Rodrigues edrodrigues@microsoft.com
Co-authored-by: Naga Bilwanth Gangarapu Naga@zensa.co
Co-authored-by: Rafael Padilha rpadilha@microsoft.com
Co-authored-by: Renato Luiz de Freitas Cunha renato.cunha@microsoft.com
Co-authored-by: Roberto de Moura Estevão Filho robertode@microsoft.com