-
Notifications
You must be signed in to change notification settings - Fork 86
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Stopping Criteria for loop #286
Conversation
experiments/mlp_mcdropout.py
Outdated
al_loop.step() | ||
if stopping_criterion.should_stop(): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wondering if you wanna change these two lines to below lines to account for general exhaustion of the dataset. The scenario does not apply to this example but as a demonstration of how to account for general exhaustion of the dataset and if the stop criteria is a metric limit
al_loop.step() | |
if stopping_criterion.should_stop(): | |
flag = al_loop.step() | |
if stopping_criterion.should_stop() or flag: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My thinking was that stopping criteria must be added to ALLoop which would check for both the criteria and exhaustion. It would require ALLoop to know about metrics which it can't do right now.
That would be a major breaking change, so we might want to have a new object instead and deprecate ALLoop 🤔 and then this new object would do the entire experiment?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
neat, Just one suggestion but not necessary.
I'll merge this and then I'll make a proposal for the new object that replaces ActiveLearningLoop into an object that can uses this new Criterion to automatically stop an experiment. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sounds good, lets go
Summary:
Allow the users to stop the loop according to various parameters.
Features:
Fixes #243
Checklist:
tests/documentation_test.py
).