Jan 27th, 2023 release: backend migration

[renatolfc]

Happy New Year, hope you had a good break! 🏂🏖️

Many of our team members took a break as well, that was one of the reason for hiatus on any updates in the repo. However, we have been working and adding lots of improvements to the codebase. 😊

With this PR, we are migrating our kubernetes cluster implementation from minikube to k3d. After running large workflows for a while now, we noticed that minikube with the Docker driver was too unstable, causing cascading crashes of both kubernetes services and our services. We've been running with k3d internally for more than a month now and it seems to be much more stable as a docker-based kubernetes platform, and we believe FarmVibes.AI users will benefit from that change.

When running the farmvibes-ai.sh script, it will automatically backup workflow state and suggest migrating a cluster from minikube to k3d.

As usual, if you already have cloned the repository, a

git pull

Should fetch the new code. We also recommend an update of the backend images with

bash farmvibes-ai.sh update-images

Finally, you might want to update the client code with

pip install ./src/vibe_core

Please let us know if you face any issues.

Apart from that, we included the following changes, listed below:

Workflows

• [🎉 NEW] Updated the index workflow to support hundreds of spectral indices. We've included support for the spyndex library, and now support all the Awesome Spectral Indices 🥳
• [🎉 NEW] Added a workflow to download forecast data using the herbie python package
• [🎉 NEW] Added workflows for downloading LIDAR/GEDI data
• [🎉 NEW] Added a workflows for downloading ERA5 monthly data
• [🎉 NEW] Added a workflow for applying ONNX models to chunks of raster images that span multiple time steps (to enable, for example, time-series analyses over raster images)
• [📈 IMPROVEMENT] We now list Sentinel-1 products from the Planetary Computer, as opposed to SciHub

Client

• [📈 IMPROVEMENT] We've improved the monitor client library to display correct durations for workflows that run for more than 24h
• [📈 IMPROVEMENT] The client now differentiates between tasks that have been queued, and tasks that have actually started running
• [📈 IMPROVEMENT] We now add the status of the workflow run to the header of the monitor method

Notebooks

• [🎉 NEW] Added a notebook for computing a timelapse of NDVI values using SpaceEye
• [🎉 NEW] Added a notebook for calculating various spectral indices (all indices listed in the Awesome Spectral Indices are supported)
• [🎉 NEW] Added a notebook that illustrates how to use the chunk_onnx workflow to run time-series analysis over raster images (in parallel)
• [📈 IMPROVEMENT] Updated the DeepMC notebooks to download recent and archived numerical weather prediction (NWP) model outputs from different cloud archive sources

Backend

• [⚒️ FIX] We fixed the workflow spec validator to check for destination ports also being a source
• [⚒️ FIX] The kubernetes backend has become much more stable due to the port to k3d
• [⚒️ FIX] We changed how we distribute work to farmvibes.ai workers, reducing the overhead of computation
• [⚒️ FIX] We now use dapr distributed locks to prevent duplication of work, making workflows run faster
• [📈 IMPROVEMENT] The history of workflow execution is now maintained between clusters
  • As a consequence, migrating from minikube to k3d will keep all the workflow data users already had
• [📈 IMPROVEMENT] The workflow validator now checks whether parameters have multiple default values
• [📈 IMPROVEMENT] We now resolve workflow parameters to infer default values and descriptions from tasks (ops)
• [📈 IMPROVEMENT] We now persist all pod logs to ~/.cache/farmvibes-ai/logs
• [📈 IMPROVEMENT] Logs are persisted in a JSON-like format
• [📈 IMPROVEMENT] We now propagate the failed and cancelled states to all ops that did not run when we cancel/fail a workflow
• [📈 IMPROVEMENT] We now support fan-out of workflow sources, enabling parallel execution of source nodes
• [📈 IMPROVEMENT] We added an ACKnowledgement message type for differentiating between started and queued tasks
• [📈 IMPROVEMENT] Allow STAC query parameters in Planetary Computer collections
• [📈 IMPROVEMENT] We've changed how tasks are assigned to workers, reducing the amount of messages interchanged between the worker and orchestrator components
• [📈 IMPROVEMENT] Added the option to skip chips that have no data when running models over large rasters