2023-04-18 release: progress bars & docs!

[renatolfc]

Hi! Over the last couple days we made changes you all are going to love! 🤩

We've merged a few documentation-related PRs which will help us have an always updated documentation site! 📚

Besides that, we've improved the client library to add progress bars to the monitor, so now you should have an idea of when ops in workflows will complete, as in the example below:

Finally, we've added improvements to the backend, generally improving workflow execution and cancellation, in addition to ensuring all our components start correctly upon cluster restarts.

As usual, if you already have cloned the repository:

git pull

should fetch the new code. We recommend an update of the library and images with:

bash farmvibes-ai.sh update

Please let us know if you have any issues.

Note ⚠️: we've identified an issue in which old clusters will fail to update to the new terraform-based format. The solution for those cases is to setup a new cluster, restoring workflow data at the end of the setup process.

---

Client:

• [📈 IMPROVEMENT] We now track the progress of individual tasks within a workflow in FarmvibesAiClient.monitor(). Subtask completion is tracked with a progress bar by default, and more detailed information about task status can be accessed with the detailed_task_info argument.

Documentation:

• [🎉 NEW] FarmVibes.AI now has documentation pages accessible through https://microsoft.github.io/farmvibes-ai/.

Workflows:

• [📈 IMPROVEMENT] Workflows now cancel/fail much faster and don't leave as many traces. Workers now drop messages from completed workflows and won't continue processing messages if the associated workflow stops running.
• [⚒️ FIX] Fixed an issue where Sentinel ingestion ops would hang until timed-out. We now use a more robust implementation that no longer spawns multiple threads to speed up downloading.

Notebooks:

• [⚒️ FIX] We removed the square root computation in notebooks/deepmc/mc_forecast.ipynb. Fixes It appears that there could be a problem with the method used for calculating the RMSE value in deepMC. #84.

Backend:

• [📈 IMPROVEMENT] Enabled Dapr injector watchdog to avoid having pods being deployed without Dapr sidecar.
• [📈 IMPROVEMENT] Remove distributed lock from workers, in favor of using acknowledgements with rabbitmq.

Known Issues:

• [⚠️] Workflows that are submitted immediately after a cluster is started (bash farmvibes-ai.sh start) will probably fail due the random initialization order of pods in kubernetes. After a couple of minutes, the cluster should stabilize and workflow execution should proceed smoothly.