Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Better SVD computation for convolution #89

Draft
wants to merge 4 commits into
base: master
Choose a base branch
from
Draft

Better SVD computation for convolution #89

wants to merge 4 commits into from

Conversation

cofri
Copy link
Collaborator

@cofri cofri commented Feb 21, 2024

  • The creation of the equivalent convolution matrix requires to remove bias and activation in the convolutional layer. This wasn't done before. This is now fixed.
  • The equivalent matrix is now sparse and the SVD computation is also based on sparse matrices. Note that the smallest singular value is very expensive to compute with scipy.sparse

@cofri cofri force-pushed the fix/bias_and_activations_in_conv_SVD branch 2 times, most recently from 161519e to 4c1d441 Compare February 22, 2024 14:49
@cofri
fix(compute_layer_sv): remove bias and activation in convolution SVD
080f4a1 
When building the equivalent matrix of a convolution layer, the bias and
activation must be removed to only keep the convolutional operation of the
layer.
By cloning the layer and setting use_bias=False and activation=None, only
the convolution is performed.
Note that cloning the model already built the new model with the correct
input shape.
@cofri
feat(compute_layer_sv): sparse SVD computation
8286dfb
@cofri
test(compute_layer_sv): add unit test with bias and activation
96eadf3 
SpectralConv2D SV computation must remove bias and activation before
computing the equivalent matrix. A unit test of a SpectralConv2D with
bias and ReLU is added.
@cofri cofri force-pushed the fix/bias_and_activations_in_conv_SVD branch from d2c6542 to 96eadf3 Compare July 26, 2024 09:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@franckma31 franckma31 franckma31 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@cofri @franckma31