Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create apply_update() fn to manage post update transformations #59

Open
bhavnicksm opened this issue Jun 15, 2024 · 0 comments
Open

Create apply_update() fn to manage post update transformations #59

bhavnicksm opened this issue Jun 15, 2024 · 0 comments
Assignees

Comments

@bhavnicksm
Copy link
Contributor

There are quite a few post update calculation transforms or modifiers that are generally useful in stabilizing the gradient descent algorithm. One such example being Stochastic Weight Averaging (SWA) and methods in the family of averaging results. The other would be lookahead.

There needs to be a general action site for these methods to act upon, so as to make it easy to extend if we need to add a new method to act, or easy to apply these to new-er optimizers that don't support it yet.

In the process of adding these updates, I don't believe we would be creating the most optimal implementation possible in memory or speed requirements, but making it usable albeit at the cost of being slower and heavier is still better than having to spend hours trying to integrate into newer optimizers.

As much as possible, I want to ensure we have it out and ready-to-use as fast as possible, and then keep the optimization of our implementation as a future work.

@bhavnicksm bhavnicksm self-assigned this Jun 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant