-
-
Notifications
You must be signed in to change notification settings - Fork 132
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to benchmark datasets with "10 times 10 fold Crossvalidation" evaluaton procedure #515
Comments
It's currently not supported. I'm not 100% sure on all the steps to be taken, but the start would be by also extracting the number of repeats from the split dimensions here, as in For proper support I think you would need to be able to specify the repeat on invocation, and also take it into account when saving and processing the results. |
@PGijsbers Thanks! The hack seems pretty straightforward. Perhaps the hack is sufficient for official implementation? I don't see a reason why we would need to treat repeats any differently from a new fold. And if we want to figure out which repeat/fold a given fold is, we can reverse engineer by looking at the evaluation procedure of the task. For example, if "5 times 2 fold Crossvalidation" is the tid's evaluation procedure, then we know "fold 5" == "third repeat, 2nd fold" (with 0-indexing this would translate to "repeat 2 fold 1") |
Without it, 100-fold CV would be indistinguishable from 10-repeated 10-fold CV (or 20-repeated 5-fold CV, or ...) without pulling the additional meta-data on the estimation procedure from OpenML. I don't really like that. It might be sufficient to just add the information to the result file(s) and use the hack otherwise. If that complicates things too much I would consider perhaps using the hack as a temporary solution on the way to full support. |
I tried benchmarking on task 168824, which is a 10 repeated 10 fold crossvalidation.
This would require 100 runs, 10 folds for each of the 10 repeats.
To do this, I made the following code edits: Innixma@994b92c
I tried setting a constraint to do this:
and created a benchmark yaml file:
However, once getting to folds above the first 10, I got the following error:
How do I make automlbenchmark work for tasks with repeated cross-validation?
The text was updated successfully, but these errors were encountered: