Multi parameter/objective in run_vizier.py #232
Conversation
|
Let's add an install script (generally actually for gpu) that includes this : ) |
|
Please create a new directory called "install_scripts" and add a script there for installing the openvizier requirements : ) |
|
To support all of the features we need
Add a prefix for things compatible with this "vizieR" e.g. "vizier_softmax_sweep.json" Handling of infeasible (maybe try catch for nan from the loss, and other types of metrics), and notes from meeting with OpenVizier |
run_vizier.py
Outdated
| study_client = clients.Study.from_study_config( | ||
| study_config, owner="owner", study_id="example_study_id" | ||
| study_config, owner="owner1", study_id="example_study_id" |
There was a problem hiding this comment.
Sounds like we should make these argparse_configurations
There was a problem hiding this comment.
please check that resuming for grid search works (with same settings).
Make it grid search doesn't repeat after stopping and restarting. Let us know results (+ or -)
run_vizier.py
Outdated
| ) | ||
|
|
||
| def evaluate_trial(loss, PPA, trial): |
There was a problem hiding this comment.
somehow we should let the model know if the loss cannot be found or is NaN to mark as infeasible.
run_vizier.py
Outdated
| study_client = clients.Study.from_study_config( | ||
| study_config, owner="owner", study_id="example_study_id" | ||
| study_config, owner="owner1", study_id="example_study_id" |
There was a problem hiding this comment.
please check that resuming for grid search works (with same settings).
Make it grid search doesn't repeat after stopping and restarting. Let us know results (+ or -)
run_vizier.py
Outdated
There was a problem hiding this comment.
Create an example script in the tests folder for default settings.
|
Refactor this into an optimization over validation loss and memory (number of bits)
Later we'll do validation loss and PPA:
|
|
Please make this into a separate script and restore the original file. Create a vizier_optimization folder, add this new script there, a readme and a sample bash script that shows how to run it. |
gkielian
changed the title
| # vizier_iterations: determine how many trails to run | ||
| # vizier_algorithm: choose which algorithm to use | ||
| # objectives: choose which objectives to use, possible choices: ["validation_loss", "num_bits"] | ||
| # c: config file, in yaml format |
There was a problem hiding this comment.
Adjust this README as per the https://www.makeareadme.com/ guide.
There was a problem hiding this comment.
Also add how to run it including sample commands (use the backticks)
There was a problem hiding this comment.
Please add a demo script here
| # vizier_iterations: determine how many trails to run | ||
| # vizier_algorithm: choose which algorithm to use | ||
| # objectives: choose which objectives to use, possible choices: ["validation_loss", "num_bits"] | ||
| # c: config file, in yaml format |
There was a problem hiding this comment.
Also add how to run it including sample commands (use the backticks)
| @@ -0,0 +1,3 @@ | |||
| #!/bin/bash | |||
| pip install google-vizier[all] | |||
There was a problem hiding this comment.
Let's specify the version to the latest
| @@ -0,0 +1,3 @@ | |||
| #!/bin/bash | |||
| pip install google-vizier[all] | |||
| pip install torch | |||
There was a problem hiding this comment.
let's omit this, should already be installed
This modified run_vizier can run multi parameter and multi objective experiments in my PC
Need to specify config file path python3 run_vizier.py --config explorations/activation_sweep.json