Confused about the setup of HPARAMS

[Vaibhavdixit02]

This is a pretty noob question, I don't have a lot of python development experience so I am probably missing something basic.

I am adapting the current implementations in the baselines directory to my new algorithm https://github.com/Vaibhavdixit02/algorithmic-efficiency/blob/sophiaalg/baselines/sophia/jax/submission.py, it is identical to the nadamw script except for the new algorithm implementation.

I keep getting an error

{'learning_rate': 0.0001, 'b1': 0.965, 'b2': 0.99, 'rho': 0.04, 'weight_decay': 0.1, 'k': 100, 'warmup_factor': 0.02}
Traceback (most recent call last):
  File "/home/gridsan/vdixit/algorithmic-efficiency/submission_runner.py", line 673, in <module>
    app.run(main)
  File "/home/gridsan/vdixit/.conda/envs/algo-eff/lib/python3.10/site-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/home/gridsan/vdixit/.conda/envs/algo-eff/lib/python3.10/site-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "/home/gridsan/vdixit/algorithmic-efficiency/submission_runner.py", line 641, in main
    score = score_submission_on_workload(
  File "/home/gridsan/vdixit/algorithmic-efficiency/submission_runner.py", line 554, in score_submission_on_workload
    timing, metrics = train_once(workload, global_batch_size,
  File "/home/gridsan/vdixit/algorithmic-efficiency/submission_runner.py", line 247, in train_once
    optimizer_state = init_optimizer_state(workload,
  File "/home/gridsan/vdixit/algorithmic-efficiency/baselines/sophia/jax/submission.py", line 153, in init_optimizer_state
    b1=hyperparameters.b1,
AttributeError: 'dict' object has no attribute 'b1'

I am wondering what I am doing wrong in how the hyperparameters should be setup?

[priyakasimbeg]

Could you attach the full submission_runner.py command and the contents of the .json file you passed in for the --tuning_search_space argument? My guess is that you don't have b1 specified in the .json file.

[Vaibhavdixit02]

The command I am using is python3 submission_runner.py --framework=jax --workload=mnist --experiment_dir=$HOME/experiments --experiment_name=my_sophia_experiment --submission_path=baselines/sophia/jax/submission.py --tuning_search_space=baselines/sophia/tuning_search_space.json the json file is here https://github.com/Vaibhavdixit02/algorithmic-efficiency/blob/sophiaalg/baselines/sophia/tuning_search_space.json

[priyakasimbeg]

I see you override the hyperparameters with a regular dict here https://github.com/Vaibhavdixit02/algorithmic-efficiency/blob/sophiaalg/baselines/sophia/jax/submission.py#L29C1-L37C2. Your code is throwing an error because you can't index in to a regular dict by using the keys as attributes.
The code you are using assumes hparams is of type configdict parsed from the .json tuning_search_space file.
Ideally you would set the feasible points in https://github.com/Vaibhavdixit02/algorithmic-efficiency/blob/sophiaalg/baselines/sophia/tuning_search_space.json. If you'd like to specify a single point for each hparam you just set 'feasible_points' to a list containing that single point.

[Vaibhavdixit02]

Ah I see. I mixed the two implementations https://github.com/mlcommons/algorithmic-efficiency/blob/main/reference_algorithms/prize_qualification_baselines/self_tuning/jax_nadamw_target_setting.py and https://github.com/mlcommons/algorithmic-efficiency/blob/main/baselines/nadamw/jax/submission.py I was using as a reference. This makes sense now! Thank you 🙌

[priyakasimbeg]

You're welcome!