π-Day Release

[lonnes]

π-Day Release

Today we are updating FarmVibes.AI with exciting new features, adding the ability to connect farm information from Microsoft Azure Data Manager for Agriculture with our existing workflows and a new farm carbon flux computation workflow whose output can be consumed by Microsoft Azure Sustainability Manager. In addition to these two new features, we are also improving and fixing bugs in several of our workflows and APIs, including a major enhancement to significantly reduce compute and storage needs in our Sentinel 2 workflows (including SpaceEye).
As usual, if you already have cloned the repository:

git pull

should fetch the new code. We recommend an update of the library and images with:

bash farmvibes-ai.sh update

Please let us know if you have any issues.

Workflows:

• [🎉 NEW] Added new workflow to get farming information from Microsoft Azure Data Manager for Agriculture. This information includes tillage, fertilization, planting, and harvest information and can be used to trigger other workflows.
• [🎉 NEW] Added a new workflow for computing Green House Gas (GHG) fluxes based on the Green House Gas Protocols and IPCC methodology with emissions factors for farm crops operations.
• [🎉 NEW] Added a new workflow that generates soil property heatmaps guided by multispectral imagery and sparse soil samples.
• [📈 IMPROVEMENT] Sentinel 2 preprocess workflow (used by all Sentinel 2 dependent workflows, including SpaceEye) now should consume half of its disk space and be twice as fast [Ability to view and edit workflows #60]. The cache for the new operation is not compatible with the cache from previous versions.
• [🎉 NEW] Added new data ingestion workflows: Sentinel 1 RTC data from Planetary Computer, GridMET and TerraClimate from Climatology Lab.
• [⚒️ FIX] Raster geometry is now correctly converted to EPSG:4326 on chunk_onnx workflow [Fixes The farm coordinates obtained from the workflow output do not match those provided as input for the same farm. #58].
• [⚒️ FIX] Sentinel 1 GRD workflows no longer crash if multiple products with the same partial prefix are available in Planetary Computer blob storage [Fixes preprocess.s1.download fails with "Found multiple prefixes matching \\'{base_pref}\\'" #59].

Notebooks:

• [🎉 NEW] Added a new notebook that describes how to integrate Green House Gas fluxes (emissions and sequestration) workflow with Microsoft Sustainability Manager.
• [🎉 NEW] Added a new notebook to calculate Normalized Vegetation Index (NDVI) for a seasonal field using Data Manager for Agriculture data.
• [🎉 NEW] Added a new notebook showing how to integrate Data Manager for Agriculture and Soil Carbon Sequestration workflow.
• [🎉 NEW] Added a notebook list with descriptions and more to come (expected duration and disk space consumed by all notebooks).
• [📈 IMPROVEMENT] Use the newly created SeasonalFieldInformation data type as input for soil carbon sequestration notebook (notebooks/carbon/whatif.ipynb).
• [⚒️ FIX] Added missing package in the DeepMC environment [Fixes mc_forecast dosen't work because historical observations depends on AGWeatherNet which doesn't work #63].

CLI:

• [🎉 NEW] We now check for available disk space before installation [connection error: connection refused to port 30000 #49].
• [📈 IMPROVEMENT] We are now deploying the cluster using terraform scripts. This will make the cluster configuration less opaque to users, and increase reliability when creating the cluster [Connection error #65].
• [📈 IMPROVEMENT] CLI now stops early if any of the required dependencies are missing.

Client

• [🎉 NEW] We added the ability to get workflow YAML files from the REST API. Check out the method client.get_workflow_yaml(workflow_name) [Ability to view and edit workflows #60].
• [📈 IMPROVEMENT] We now have better error messages for when workflow runs fail.
• [⚒️ FIX] We restored a behavior where if a resource already exists, we log the issue but don’t stop the workflow execution [spaceeye.spaceeye.spaceeye in crop segementation notebook crashes the cluster  #56].
• [🎉 NEW] The client now checks with the REST API whether there is space available in the cache when submitting new workflows, if space is low (< 50 GiB), the client now issues a warning to users.

Backend

• [🎉 NEW] The REST-API now outputs the complete YAML definition of workflows at the endpoint /v0/workflows/{workflow_name}?return_format=yaml, where workflow_name is the name of the workflow [Ability to view and edit workflows #60].
• [🎉 NEW] We added a new endpoint to the REST API that outputs system usage metrics of the node currently hosting the REST API.

Known Issues

• [🐛 BUG] The workflow run monitor might display incorrect times (including negative ones) depending on time zone differences between the client’s clock and the server’s clock.
• [🐛 BUG] On rare occasions, when we download a corrupted Sentinel 2 GeoTIFF from the Planetary Computer, the stacking of Sentinel 2 bands might fail and leave the operation hanging. If that happens, s2 downloads will halt, and will require a restart of the cluster, which can be done with bash farmvibes-ai.sh restart, and resubmission of the workflow.