Improved logging and added new data ingestion workflows

[renatolfc]

Here's a summary of the changes from the last pull request:

Backend:

• [🎉 NEW] We enhanced the logging capabilities of FarmVibes.AI, providing more control over log file management, file size restrictions and log rotation. When setting up a new cluster with farmvibes-ai <local/remote> setup, users can pass the --max-log-file-bytes argument to define the maximum number of bytes for a log file, and the --log-backup-count that determines the number of log files to keep per service (e.g., orchestrator, cache, data-ops) with old log files being deleted. The default values for these arguments are 100MB and 5, respectively.
• [📈 IMPROVEMENT] We implemented several enhancements in our caching service increasing the speed in which cached results are recovered.
• [📈 IMPROVEMENT] We added a restart subcommand to farmvibes-ai remote, allowing AKS remote clusters to be restarted from the CLI.
• [⚒️ FIX] We fixed a few encoding bugs that would break the setup process on Windows machines. Related to issue Error installing a Remote Cluster in Azure #136.

Workflows:

• [🎉 NEW] We added a new workflows/data_ingestion/weather/download_herbie workflow for downloading numerical weather prediction from Herbie.
• [🎉 NEW] We added new data ingestion workflows to download Forest Extent data from Global Land Analysis & Discovery (GLAD). workflows/data_ingestion/glad/glad_forest_extent_download downloads individual tiles and workflows/data_ingestion/glad/glad_forest_extent_download_merge provides the merged rasters.
• [🎉 NEW] We added new data ingestion workflows to download global 25m resolution SAR mosaics and forest/non-forest maps from ALOS Forest/Non-Forest Annual Mosaic. Similarly, workflows/data_ingestion/alos/alos_forest_extent_download downloads individual tiles and workflows/data_ingestion/alos/alos_forest_extent_download_merge provides the merged rasters.
• [🎉 NEW] We added new data ingestion workflows for downloading basemaps from Bing Maps. Similarly, workflows/data_ingestion/bing/basemap_download downloads multiple BingMaps tiles separately, and workflows/data_ingestion/bing/basemap_download_merge merge the individual tiles into a single image.
• [📈 IMPROVEMENT] We extended the Segment Anything Model (SAM) workflow (workflows/farm_ai/segmentation/segment_basemap) to process basemap inputs, allowing users to segment maps with higher resolution than the previous version that operated on Sentinel-2 images.
• [⚒️ FIX] We fixed an issue with the workflows/data_processing/heatmap/classification workflow that would cause the output heatmap to have a different range of values than the input data. Related to Discrepancy in Units between Input Sensor Data and Heatmap Output for Humidity and Salinity Properties. #134.

Notebooks:

• [🎉 NEW] We added three notebooks that perform field-level analysis, combining field boundary detection with index computation (notebooks/sentinel/field_level_spectral_indices.ipynb), irrigation classification (notebooks/irrigation/field_level_irrigation_classification.ipynb) and optimal sensor placement (notebooks/sensor/optimal_locations_segmentation.ipynb).
• [🎉 NEW] We added a notebook that performs SAM segmentation over BingMaps basemaps (notebooks/segment_anything/basemap_segmentation.ipynb).

Documentation:

• [📈 IMPROVEMENT] We improved our Remote Cluster User Guide, adding details on the requirements expected from the user's role, subscription, and machine for setting up a remote cluster.

---

Co-authored-by: Bruno Silva brunosilva@microsoft.com
Co-authored-by: Eduardo Rodrigues edrodrigues@microsoft.com
Co-authored-by: Jessica Wolk Jessica.Wolk@microsoft.com
Co-authored-by: Naga Bilwanth Gangarapu Naga@zensa.co
Co-authored-by: Rafael Padilha rpadilha@microsoft.com
Co-authored-by: Renato Luiz de Freitas Cunha renato.cunha@microsoft.com
Co-authored-by: Rick Gutierrez ricardgu@microsoft.com