Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor Optimizer Config API #45

Open
bhavnicksm opened this issue Jun 1, 2024 · 2 comments
Open

Refactor Optimizer Config API #45

bhavnicksm opened this issue Jun 1, 2024 · 2 comments
Assignees
Labels
enhancement New feature or request wontfix This will not be worked on

Comments

@bhavnicksm
Copy link
Contributor

Currently every optimizer comes with a config specifically for that optimizer that manages the hyperparameters for the optimizer.

This is made because of the following reasons:

  • A lot of hyper-parameters need to transfer properly with Inheritance, which is essential for the composability of the optimizers. By having the hyper-parameters inside a Dataclass, we can easily append these Dataclasses by only the new additional parameters we might need, along with the defaults that are useful.
  • Ease of transfer and storage: Keep all Optimizer hyper-params in one place, transform them as a dict, or as a yaml, save it, load it. This increases the replication ability of these optimizers. Hyper-parameters need to be saved, just like the optimizer state is saved.

This diverges from the PyTorch-esque way optimizers are where the classes need to be provided with full information on all the parameters.

This issue is to find a decent middle ground between the two -> Allow for immediate tiny changes in the Hyperparameters on the spot, with the object, while also allowing for inheritable parameters.

This adds a significant overhead and makes things a bit more complex, so it should ideally be abstracted out in the BaseConfig and BaseOptimizer, with all the utility functions that transform from one format to the other.

@bhavnicksm bhavnicksm self-assigned this Jun 1, 2024
@bhavnicksm bhavnicksm added the enhancement New feature or request label Jun 1, 2024
@bhavnicksm
Copy link
Contributor Author

The reason why Config classes were originally set was because I expected all the optimizers to have a lot of "features" or options to tune in the future which makes the constructors really long.

Imagine the complexity if each new optimizer has to mention 30+ arguments in their init signature. Sounds tough!

@bhavnicksm bhavnicksm added the wontfix This will not be worked on label Jun 15, 2024
@bhavnicksm
Copy link
Contributor Author

This issue is related to #62

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request wontfix This will not be worked on
Projects
None yet
Development

No branches or pull requests

1 participant