-
-
Notifications
You must be signed in to change notification settings - Fork 143
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cache run predictions #1191
Labels
Comments
After an offline discussion with @PGijsbers we agreed that this should be an optional feature, i.e. that caching is disabled by default, but can be enabled. |
There are definitely cases where this is useful (experimenting with evaluation metrics or ensembling), but the average user that probably doesn't load the same runs many times. Because it would quickly occupy a lot of disk space, we think opt-in is better. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Problem
Predictions of runs are not cached when downloaded. Note that predictions only get downloaded when
get_metric_fn
is called in the first place (this is desired behavior, the description file already contains precomputed evaluations).MWE
CLI:
ls ~/.openml/org/openml/www/runs/10591753/
Output:
ls: /Users/pietergijsbers/.openml/org/openml/www/runs/10591753/: No such file or directory
Execute:
output:
CLI:
ls ~/.openml/org/openml/www/runs/10591753/
Output:
description.xml
Note that there are no signs of the prediction arff file being present on disk - as you would expect from reading the source code.
The text was updated successfully, but these errors were encountered: