Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ADD] Support for KFAC with type-2 Fisher #56

Merged
merged 18 commits into from
Nov 9, 2023
Merged

Conversation

runame
Copy link
Collaborator

@runame runame commented Nov 7, 2023

Resolves #49.

Note: I can't immediately see anymore why the implementation for CrossEntropyLoss works like this, will have to think about it, but should be correct.

@runame runame added the enhancement New feature or request label Nov 7, 2023
@runame runame requested a review from f-dangel November 7, 2023 21:57
@runame runame linked an issue Nov 7, 2023 that may be closed by this pull request
Copy link

github-actions bot commented Nov 7, 2023

Pull Request Test Coverage Report for Build 6802629667

  • 35 of 37 (94.59%) changed or added relevant lines in 1 file are covered.
  • 40 unchanged lines in 2 files lost coverage.
  • Overall coverage decreased (-4.5%) to 88.033%

Changes Missing Coverage Covered Lines Changed/Added Lines %
curvlinops/kfac.py 35 37 94.59%
Files with Coverage Reduction New Missed Lines %
curvlinops/examples/utils.py 6 41.67%
curvlinops/fisher.py 34 28.85%
Totals Coverage Status
Change from base Build 6801631981: -4.5%
Covered Lines: 743
Relevant Lines: 844

💛 - Coveralls

Conflicts:
	curvlinops/kfac.py
	test/test_kfac.py
curvlinops/kfac.py Outdated Show resolved Hide resolved
@f-dangel
Copy link
Owner

f-dangel commented Nov 8, 2023

I decided to start a section 'Internals' in the documentation which explains the tricky parts in math and also contains links to derivations. This might be useful for others who are interested in extending this library, or learning more about the details of each curvature approximation.

I have worked on this topic for quite a bit, so it would be great to get feedback if it is comprehensible.

You can take a look here.

Copy link
Collaborator Author

@runame runame left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like the refactor in terms of backpropagating the square root loss Hessian and the idea of having an internals section in the docs! I left a few comments/questions.

curvlinops/kfac_utils.py Outdated Show resolved Hide resolved
curvlinops/kfac_utils.py Show resolved Hide resolved
curvlinops/kfac.py Show resolved Hide resolved
curvlinops/kfac.py Show resolved Hide resolved
curvlinops/kfac_utils.py Show resolved Hide resolved
curvlinops/kfac_utils.py Outdated Show resolved Hide resolved
curvlinops/kfac_utils.py Show resolved Hide resolved
@f-dangel f-dangel merged commit 1141c50 into development Nov 9, 2023
12 of 13 checks passed
@f-dangel f-dangel deleted the kfac-type-2 branch November 9, 2023 15:16
@runame runame mentioned this pull request Nov 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support type-2 Fisher in KFAC
2 participants