Initialization seed loop in BaseStrategy

[Mattdl]

CL experiments don't often take very long. For computer clusters it's easier to have one job being scheduled that runs longer (e.g. 1 job that runs the 5 seeds, instead of having to wait for 5 jobs to get scheduled).
Can this be integrated in the BaseStrategy? And is it possible to accumulate evaluation statistics over different seeds?

[vlomonaco]

At the moment we do not support hyperparams selection or sophisticated metrics results accumulation (the user should implement them from scratch), we should think how to support them better (maybe using orion?)

@mmasana I'm sure may offer more insights on this issue :)

[mmasana]

We used a GridSearch system, similar to the one in the TIL survey with @Mattdl. However, we implemented it ourselves, so using smth like Orion could be interesting to make it more standardized. For each method to define which hyperparameters are important to be searched and which are more or less set, can be a rabbit hole though.

I would be in favour of having some kind of accumulation of statistics over runs with the same parameters but different seed. As of now, we solve it as mentioned, by having each job run multiple seeds (parallel or series, depending on resources). Then we have separate scripts that read the output logs and generate graphs, tables or whatever needed.

[Mattdl]

Yes I agree, I think the ad-hoc solution might be easiest and certainly useful!

[AntonioCarta]

I would keep training and model selection separated, like scikit-learn does. Something like GridSearch(param_grid, model_class, train_stream, val_stream, test_stream), where model_class is the strategy class. Grid search covers @Mattdl usecases when you have a single configuration.
I think this is a good contribution for the evaluation module. It's also a good first issue.

Keep in mind that as of now we don't have a method to split data streams yet, therefore you can't separate the train stream intro train and validation. This will be implemented soon (hopefully).