Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

What-if scenario evaluation for carbon sequestration notebook #151

Closed
Click2cloud-Abhijeet opened this issue Feb 19, 2024 · 12 comments
Closed

What-if scenario evaluation for carbon sequestration notebook #151

Click2cloud-Abhijeet opened this issue Feb 19, 2024 · 12 comments
Assignees
@v-ngangarapu
Labels
bug Something isn't working workflows Issues encountered when running workflows

Comments

@Click2cloud-Abhijeet
Copy link

Click2cloud-Abhijeet commented Feb 19, 2024 •
edited by rafaspadilha
Loading

Hi team ,
Hope you are doing good !!!

I was trying to run a "whatif.ipynb" notebook in new Farmvibes-ai cluster , but Notebook is getting failed.
When I checked using run.reason it is giving me below reason.

RuntimeError: 
Failed to run op whatif_comet_op in workflow run id a1225b9a-1e6b-4e9a-a8a3-e5d4b1a2bb12 for input with message id 00-a1225b9a1e6b4e9aa8a3e5d4b1a2bb12-fc18cd0aed4b24db-01. 

Error description: 
<class \'RuntimeError\'>:  PyngrokNgrokError("The ngrok process errored on start: {}.".format(ngrok_process.startup_error),
pyngrok.exception.PyngrokNgrokError: 

The ngrok process errored on start: Your ngrok-agent version "2.3.41" is too old. The minimum supported agent version for your account is "3.2.0". Please update to a newer version with `ngrok update`, by downloading from https://ngrok.com/download, or by updating your SDK version. Paid accounts are currently excluded from minimum agent version requirements. To begin handling traffic immediately without updating your agent, upgrade to a paid plan: https://dashboard.ngrok.com/billing/subscription.

ERR_NGROK_121.

I am also providing a Screen shot of an error.
Please help me to find the solution for it.
Thank you
whatif_error
@github-actions github-actions bot added the triage Issues still not triaged by team label Feb 19, 2024
@rafaspadilha rafaspadilha added notebooks Issues encountered while running the notebooks bug Something isn't working workflows Issues encountered when running workflows and removed notebooks Issues encountered while running the notebooks labels Feb 19, 2024
@v-ngangarapu
Copy link
Collaborator

Hi Abhijit,
Thanks for reaching us with your observations. I was able to reproduce the same results you're seeing.

Will do more research on our end and let you know the possible solutions.

Thanks

@v-ngangarapu v-ngangarapu self-assigned this Feb 19, 2024
@rafaspadilha rafaspadilha removed the triage Issues still not triaged by team label Feb 20, 2024
@v-ngangarapu
Copy link
Collaborator

Hi
The issue is happening due to recent upgrade done to ngrok package. It's resolved now.
Will be pushing the change in next build.

Thanks

@Click2cloud-Abhijeet
Copy link
Author

Hi The issue is happening due to recent upgrade done to ngrok package. It's resolved now. Will be pushing the change in next build.

Thanks

Thank you for your response and happy to hear that.

@gmesserman
Copy link

gmesserman commented Feb 26, 2024 •
edited
Loading

Hello,

When do you expect this to be pushed and become available?

Thanks,
GIl

@v-ngangarapu
Copy link
Collaborator

Hi Gil,
It should be pushed soon, I don't have the exact date.

Thanks

@rafaspadilha rafaspadilha added the next-release-fix Issues with a fix planned for the next release label Mar 6, 2024
@kennrichie
Copy link

Hi Gil, It should be pushed soon, I don't have the exact date.

Thanks

Thanks for the information, but I got the same issue today, 22nd of March, weeks after

@imanema1
Copy link

me too

@arewtech
Copy link

Is this finished already? I also found the same thing as above 😭

@rafaspadilha
Copy link
Contributor

Hi, folks. We are planning a release for this week with a fix for this issue. Thank you for your patience.

@rafidhasan448
Copy link

When will this issue be solved ?

@renatolfc renatolfc mentioned this issue Apr 5, 2024
Merged
@rafaspadilha rafaspadilha removed the next-release-fix Issues with a fix planned for the next release label Apr 5, 2024
@rafaspadilha
Copy link
Contributor

rafaspadilha commented Apr 5, 2024 •
edited
Loading

Folks, we just released a new version of FarmVibes.AI (PR #157) that updates ngrok version and should fix this issue.
Please, let us know if you are able to test it and if any more issue arises.

renatolfc added a commit that referenced this issue Apr 5, 2024
@renatolfc
FarmVibes.AI 24.04 (#157)
a1cad6a 
This is a sync of our repo. Here's the summary of the changes:

## Support to new ADMA API version:
**[💔 BREAKING CHANGES]** We added support to the new [Azure Data Manager
for Agriculture
(ADMA)](https://learn.microsoft.com/en-us/rest/api/data-manager-for-agri/?view=rest-data-manager-for-agri-dataplane-2023-11-01-preview)
API version _2023-11-01-preview_. To comply with the new API, some of
our data types, workflows, and notebooks were updated. A summary of the
changes are:

- The `farmer_id` property has now changed to `party_id` in
[`ADMAgSeasonalFieldInput`](https://microsoft.github.io/farmvibes-ai/docfiles/code/vibe_core_data/farm.html#vibe_core.data.farm.ADMAgSeasonalFieldInput)
object (which is passed as input for our workflows) and other [ADMA data
types](https://microsoft.github.io/farmvibes-ai/docfiles/code/vibe_core_data/farm.html).
Also, there is no more the `boundary_id` field as boundary resources are
no more single objects in ADMA. The boundary information is part of the
ADMA entities (seasonal field, field etc.).
- All operations (seasonal field, field, prescriptions etc.) now have
their own geometry.
- Planting information in Seasonal Field was moved to a new operation
["Planting
Data"](https://learn.microsoft.com/en-us/rest/api/data-manager-for-agri/dataplane/planting-data?view=rest-data-manager-for-agri-dataplane-2023-11-01-preview).
- The affected workflows are:
[`data_ingestion/admag/admag_seasonal_field`](https://microsoft.github.io/farmvibes-ai/docfiles/markdown/workflow_yaml/data_ingestion/admag/admag_seasonal_field.html),
[`data_ingestion/admag/prescriptions`](https://microsoft.github.io/farmvibes-ai/docfiles/markdown/workflow_yaml/data_ingestion/admag/prescriptions.html),
and
[`farm_ai/agriculture/heatmap_using_classification_admag`](https://microsoft.github.io/farmvibes-ai/docfiles/markdown/workflow_yaml/farm_ai/agriculture/heatmap_using_classification_admag.html).
- Likewise, the following notebooks have been updated:
[notebooks/admag](https://github.com/microsoft/farmvibes-ai/tree/main/notebooks/admag)
and
[notebooks/heatmaps/nutrients_using_classification_admag](https://github.com/microsoft/farmvibes-ai/blob/main/notebooks/heatmaps/nutrients_using_classification_admag.ipynb).


## Workflows:
- [🎉 NEW] We added new workflows to download and process forest extent
data. These workflows include: (i) the Advanced Land Observing Satellite
(ALOS) forest/non-forest map, (ii) the Global Land Analysis (GLAD)
forest map, and (iii) the Global Forest Change (Hansen) datasets. The
processing workflow detects forest changes using statistical tests
(specifically, the Cochran-Armitage test) to determine whether the
change represents a trend.
- [📈 IMPROVEMENT] We are releasing an updated version of the SpaceEye
workflow that leverages Sentinel-1 RTC products instead of GRD products
with SNAP preprocessing. These changes lead to a significant speedup and
a decrease in disk space consumption. Faster, lighter, and cloudless-er
images to y'all! Due to this new version of the workflow, cached results
are invalid and will be recomputed on new runs.
- [📈 IMPROVEMENT] We extended the
`data_ingestion/weather/download_herbie` workflow to allow downloading
both analysis (zero lead time) up to a point in time and then forecast
with progressively increasing lead times after that.
- [⚒️ FIX] We added a [script](todo-add-link-to-script) to convert
Segment Anything Model (SAM) weights to ONNX format and import it into
the local FarmVibes.AI cluster to be used in the SAM workflows and
notebooks. For instructions on how to use the script, refer to the
_"Adding SAM's ONNX models to the cluster"_ entry in our
[Troubleshooting
page](https://microsoft.github.io/farmvibes-ai/docfiles/markdown/TROUBLESHOOTING.html).
Related to issues #145 and #153.
- [💔 **BREAKING CHANGES**] Planetary Computer API key is required for
all workflows that download or use Sentinel-1 products (e.g.,
`data_ingestion/sentinel1/preprocess_s1` and
`data_ingestion/spaceeye/spaceeye`). The documentation with the
parameter description will be updated in a future release.

## Backend:
- [📈 IMPROVEMENT] We updated our base images to use Python `3.11`, as
well as DAPR `1.13.0`, fixing a few security vulnerabilities of the
previous version that could lead to DoS in the kubernetes services.
Additionally, our DAPR logs are now in JSON format.
- [📈 IMPROVEMENT] We updated `ngrok` package to version `7.1.2`, which
will solve the issues with the [What-if scenario evaluation for carbon
sequestration](https://github.com/microsoft/farmvibes-ai/blob/main/notebooks/carbon/whatif.ipynb)
notebook and `farm_ai/carbon_local/carbon_whatif` workflow. Related to
issue #151.
- [📈 IMPROVEMENT] We improved our worker retry policy and are now using
exponential backoff for retries. This means workers will be less idle on
long queues, avoiding cases where workers would wait for one minute
before starting a queued task.
- [⚒️ FIX] We fixed an issue with the `add-secret` subcommand not adding
secrets to the KeyVault during remote (AKS) cluster setup. Related to
issue #147.

## Notebooks:
- [🎉 NEW] We added a few notebooks showcasing the new forest workflows,
such as downloading [ALOS](todo-add-link-to-notebook) and
[GLAD](todo-add-link-to-notebook) forest maps, and [evaluating forest
changes over time with the Hansen maps](todo-add-link-to-notebook). We
also provide a notebook highlighting how to detect increase/decrease
trends in forest pixel levels over an user-input geometry and time
range.

## Documentation:
- [🎉 NEW] We added a new [REST API documentation
page](https://microsoft.github.io/farmvibes-ai/docfiles/markdown/REST_API.html)
describing the endpoints, parameters, and responses for our API.


## Client:
- [📈 IMPROVEMENT & ⚒️ FIX] We made a few quality-of-life improvements
and bug fixes to our python client. Users can now monitor the last `n`
runs with `client.monitor(n)`, which will print a summarized table with
their progress, as well as retrieve the last `n` `VibeWorkflowRun`
objects with `client.get_last_runs(n)` method. We also added minor bug
fixes to the `monitor` and `document_workflow` methods, which were
breaking for deleated runs and workflows with None/empty value
parameters, respectivelly.

---------
Co-authored-by: Bruno Silva <brunosilva@microsoft.com>
Co-authored-by: Eduardo Rodrigues <edrodrigues@microsoft.com>
Co-authored-by: Leonardo Nunes <lnunes@microsoft.com>
Co-authored-by: Naga Bilwanth Gangarapu <Naga@zensa.co>
Co-authored-by: Rafael Padilha <rpadilha@microsoft.com>
Co-authored-by: Renato Luiz de Freitas Cunha
<renato.cunha@microsoft.com>
Co-authored-by: Roberto de Moura Estevão Filho <robertode@microsoft.com>
@rafaspadilha
Copy link
Contributor

Closing this for now. Please, let us know if you still have any issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@v-ngangarapu v-ngangarapu

Labels
bug Something isn't working workflows Issues encountered when running workflows
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@rafaspadilha @gmesserman @arewtech @v-ngangarapu @Click2cloud-Abhijeet @rafidhasan448 @imanema1 @kennrichie