Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add AdEMAMix Optimizer #1057

Closed
wants to merge 1 commit into from

Conversation

mathDR
Copy link

@mathDR mathDR commented Sep 13, 2024

This PR adds the AdaMAMix optimizer from The arxiv preprint: THE ADEMAMIX OPTIMIZER:
BETTER, FASTER, OLDER

Closes #1058

The docs have been updated, along with the relevant files. Furthermore, I ran a similar "test" to replicate the Rosenbrock figure from the paper:
image

Currently the docstrings are implemented, but further descriptions should/could be added. I will reach out to the paper authors to assist with that (if they are willing).

@vroulet
Copy link
Collaborator

vroulet commented Sep 14, 2024

Thanks @mathDR for the contribution!
Optimizers that have not passed the test of time (too recent like this one) are put in the contrib folder (see https://optax.readthedocs.io/en/latest/development.html#inclusion-criteria).
Before continuing (making this optimizer in the contribs folder), can you wait for #1060 to be merged?
I would like all optimizers to abide to common tests to avoid issues like #1038.
PS: use an editor that enforces indents of two spaces not four.

@mathDR
Copy link
Author

mathDR commented Sep 16, 2024

Okay thanks for the comments @vroulet. I will put an alert to check the status of #1060 and see when that is merged.

@vroulet
Copy link
Collaborator

vroulet commented Sep 24, 2024

Hello @mathDR
#1060 has been merged.
If you want you can continue this PR. Put the algorithm in the contrib folder and make sure it passes the common tests.

@mathDR mathDR closed this by deleting the head repository Sep 26, 2024
@mathDR
Copy link
Author

mathDR commented Sep 26, 2024

I closed this and deleted the forked repo (will be easier to fork again and start anew)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature request for the AdeMAMix optimizer.
2 participants