Support for logging multiple runs and comparison between them (model selection)

[AntonioCarta]

I think the logging module right now is at a good point. We need to polish some of the outputs but overall it si clear and informative. One thing I'm missing though is some kind of automation for multiple experiment management.

I'm thinking about two main use cases:

• model selection, where I do a sweep of some hyperparameters
• limited exploratory search, where I manually provide the hyperparameters (typically by command line arguments) and run the same script multiple times

Right now I can do this but I need to be careful to save each experiment in a different folder. It is also cumbersome to compare different runs without external tools. Since Avalanche provides logging features, I would like to have minimal support for:

• Automatic selection and creation of a new folder. I give a name to the experiment (core50_replay) and the logging tools automatically save in each run in a separate folder (core50_replay/run0, core50_replay/run1).
• Automatic collection of hyperparameters. Probably at this stage I would limit to the collection of command-line arguments by argparse (maybe also the seeds for reproducibility?).
• Possibly a report at the root level core50_replay where I can quickly compare the different runs. As a starting point the report can be really basic like a table where each row has the name of the run (run0) and the main metric value (stream accuracy).

I think W&B has everything we need to provide a minimal support for these use cases.

[AndreaCossu]

Yes I agree. This should not be difficult. I have implemented similar functionalities in my projects using Avalanche and it is quite straightforward.