Add ADAM optimizer

[arvigj]

No description provided.

[codecov]

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (e9bb718) 88.95% compared to head (49edbec) 89.12%.

Files	Patch %	Lines
...rc/polysolve/nonlinear/descent_strategies/ADAM.cpp	95.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #57      +/-   ##
==========================================
+ Coverage   88.95%   89.12%   +0.16%     
==========================================
  Files          47       49       +2     
  Lines        1902     1950      +48     
==========================================
+ Hits         1692     1738      +46     
- Misses        210      212       +2     

Flag	Coverage Δ
polysolve	89.12% <95.83%> (+0.16%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[teseoch]

wow that was quick. Does Adam always produce a descent direction?

[arvigj]

Yes, but the descent direction may not be in the direction of the gradient, so then the descent strategy would change. I don't know if this strategy is good, just wanted to put it here for other to experiment with.

[teseoch]

can you add the stochastic flag here too?

do we need the is_direction_descent?

[Huangzizhou]

If Adam doesn't necessarily produce a descent direction, you need

bool is_direction_descent() override { return false; }

in the class.