Experiment/Trial state is ephemeral

[johnaohara]

From discussion on #39

"In optuna we aren't using persistent storage at present"

At present all experiment / trial config data is stored in memory and not in any form of persistent storage. This creates a number of challenges;

1. If HPO is running as a service (esp. in a K8's environment) any containers will be ephemeral. This means that K8s can decide to restart pods, current experiments would be lost after a restart and experiments would have to be restarted
2. There is no persistent history of experiments, trials or results.
3. There is no ability to restart an experiment that was already under-way. This will currently be difficult to restart due to the synchronous architecture of the optuna library.

Are there plans to introduce persistent storage of experiment/trial data?

[johnaohara]

For info, I am handling these issues (apart from no. 3) in my HPO project as I need to track state for multiple experiments simultaneous and asynchronously, but it might be worthwhile to investigate if this functionality fits with the plans for HPO

[khansaad]

Hi @johnaohara, Thanks for creating this issue.

We were having an internal discussion about it and I have created a discussion idea #45 where I have shared a basic DB design for the same. We can discuss about it there and suggest ideas to move forward.

[johnaohara]

@khansaad thanks for the update