Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement Spatial Transformer used by LDM UNet #438

Closed
wants to merge 1 commit into from
Closed

Implement Spatial Transformer used by LDM UNet #438

wants to merge 1 commit into from

Conversation

abhinavarora
Copy link
Contributor

Summary:
TL;DR
Implements the Spatial Transformer and the Cross-Attention layers used within the LDM Unet.

Summary of Changes

  1. Implement TransformerCrossAttentionLayer similar to LDM;s implementation. This module uses GEGLU activation instead of GLU
  2. Implements SpatialTransformer that operates on images.
  3. Implement GEGLU activation
  4. Implement zero_module util
  5. Add tests for each component

Differential Revision: D47709486

@facebook-github-bot facebook-github-bot added CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. fb-exported labels Jul 24, 2023
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D47709486

@codecov-commenter
Copy link

codecov-commenter commented Jul 24, 2023 •
edited
Loading

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.05% 🎉

Comparison is base (82c1dc2) 68.66% compared to head (a5c8937) 68.72%.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #438      +/-   ##
==========================================
+ Coverage   68.66%   68.72%   +0.05%     
==========================================
  Files         169      169              
  Lines       11355    11376      +21     
==========================================
+ Hits         7797     7818      +21     
  Misses       3558     3558
Files Changed Coverage Δ
tests/modules/layers/test_activation.py 100.00% <100.00%> (ø)
tests/utils/test_common.py 100.00% <100.00%> (ø)
torchmultimodal/modules/layers/activation.py 100.00% <100.00%> (ø)
torchmultimodal/utils/common.py 92.39% <100.00%> (+0.25%) ⬆️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

abhinavarora pushed a commit to abhinavarora/multimodal that referenced this pull request Jul 25, 2023
@facebook-github-bot
Implement Spatial Transformer used by LDM UNet (facebookresearch#438)
f0b21d6 
Summary:
Pull Request resolved: facebookresearch#438

**TL;DR**
Implements the Spatial Transformer and the Cross-Attention layers used within the LDM Unet.

**Summary of Changes**
1. Implement `TransformerCrossAttentionLayer` similar to LDM;s implementation. This module uses `GEGLU` activation instead of `GLU`
1. Implements `SpatialTransformer` that operates on images.
1. Implement `GEGLU` activation
1. Implement `zero_module` util
1. Add tests for each component

Differential Revision: D47709486

fbshipit-source-id: 61270e20a70097bb569392eec116b2b0430297df
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D47709486

abhinavarora pushed a commit to abhinavarora/multimodal that referenced this pull request Jul 25, 2023
@facebook-github-bot
Implement Spatial Transformer used by LDM UNet (facebookresearch#438)
98cf9d3 
Summary:
Pull Request resolved: facebookresearch#438

**TL;DR**
Implements the Spatial Transformer and the Cross-Attention layers used within the LDM Unet.

**Summary of Changes**
1. Implement `TransformerCrossAttentionLayer` similar to LDM;s implementation. This module uses `GEGLU` activation instead of `GLU`
1. Implements `SpatialTransformer` that operates on images.
1. Implement `GEGLU` activation
1. Implement `zero_module` util
1. Add tests for each component

Differential Revision: D47709486

fbshipit-source-id: 156e3d08b8ade3d47faae8a481cdcdaadc581887
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D47709486

abhinavarora pushed a commit to abhinavarora/multimodal that referenced this pull request Jul 25, 2023
@facebook-github-bot
Implement Spatial Transformer used by LDM UNet (facebookresearch#438)
0ce7869 
Summary:
Pull Request resolved: facebookresearch#438

**TL;DR**
Implements the Spatial Transformer and the Cross-Attention layers used within the LDM Unet.

**Summary of Changes**
1. Implement `TransformerCrossAttentionLayer` similar to LDM;s implementation. This module uses `GEGLU` activation instead of `GLU`
1. Implements `SpatialTransformer` that operates on images.
1. Implement `GEGLU` activation
1. Implement `zero_module` util
1. Add tests for each component

Differential Revision: D47709486

fbshipit-source-id: b4945c6401b302503212d014767a827f4ac90c0b
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D47709486

abhinavarora pushed a commit to abhinavarora/multimodal that referenced this pull request Jul 26, 2023
@facebook-github-bot
Implement Spatial Transformer used by LDM UNet (facebookresearch#438)
be509fb 
Summary:
Pull Request resolved: facebookresearch#438

**TL;DR**
Implements the Spatial Transformer and the Cross-Attention layers used within the LDM Unet.

**Summary of Changes**
1. Implement `TransformerCrossAttentionLayer` similar to LDM;s implementation. This module uses `GEGLU` activation instead of `GLU`
1. Implements `SpatialTransformer` that operates on images.
1. Implement `GEGLU` activation
1. Implement `zero_module` util
1. Add tests for each component

Differential Revision: D47709486

fbshipit-source-id: 5a0fb0faa8f85c4c9b01b72744ac77bbe33650a7
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D47709486

abhinavarora pushed a commit to abhinavarora/multimodal that referenced this pull request Jul 26, 2023
@facebook-github-bot
Implement Spatial Transformer used by LDM UNet (facebookresearch#438)
caacfbd 
Summary:
Pull Request resolved: facebookresearch#438

**TL;DR**
Implements the Spatial Transformer and the Cross-Attention layers used within the LDM Unet.

**Summary of Changes**
1. Implement `TransformerCrossAttentionLayer` similar to LDM;s implementation. This module uses `GEGLU` activation instead of `GLU`
1. Implements `SpatialTransformer` that operates on images.
1. Implement `GEGLU` activation
1. Implement `zero_module` util
1. Add tests for each component

Differential Revision: D47709486

fbshipit-source-id: c0419ec42f07f669eaa524a9e295bd1d51708f3e
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D47709486

abhinavarora pushed a commit to abhinavarora/multimodal that referenced this pull request Jul 26, 2023
@facebook-github-bot
Implement Spatial Transformer used by LDM UNet (facebookresearch#438)
cc82c3f 
Summary:
Pull Request resolved: facebookresearch#438

**TL;DR**
Implements the Spatial Transformer and the Cross-Attention layers used within the LDM Unet.

**Summary of Changes**
1. Implement `TransformerCrossAttentionLayer` similar to LDM;s implementation. This module uses `GEGLU` activation instead of `GLU`
1. Implements `SpatialTransformer` that operates on images.
1. Implement `GEGLU` activation
1. Implement `zero_module` util
1. Add tests for each component

Differential Revision: D47709486

fbshipit-source-id: dbb670e4510451c48a88acc05259a03c20a85b6d
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D47709486

abhinavarora pushed a commit to abhinavarora/multimodal that referenced this pull request Jul 27, 2023
@facebook-github-bot
Implement Spatial Transformer used by LDM UNet (facebookresearch#438)
7d7e9cd 
Summary:
Pull Request resolved: facebookresearch#438

**TL;DR**
Implements the Spatial Transformer and the Cross-Attention layers used within the LDM Unet.

**Summary of Changes**
1. Implement `TransformerCrossAttentionLayer` similar to LDM;s implementation. This module uses `GEGLU` activation instead of `GLU`
1. Implements `SpatialTransformer` that operates on images.
1. Implement `GEGLU` activation
1. Implement `zero_module` util
1. Add tests for each component

Reviewed By: pbontrager

Differential Revision: D47709486

fbshipit-source-id: 3eb7164a25a1726911cba65b19528cc9753a740d
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D47709486

@facebook-github-bot
Implement Spatial Transformer used by LDM UNet (facebookresearch#438)
a5c8937 
Summary:
Pull Request resolved: facebookresearch#438

**TL;DR**
Implements the Spatial Transformer and the Cross-Attention layers used within the LDM Unet.

**Summary of Changes**
1. Implement `TransformerCrossAttentionLayer` similar to LDM;s implementation. This module uses `GEGLU` activation instead of `GLU`
1. Implements `SpatialTransformer` that operates on images.
1. Implement `GEGLU` activation
1. Implement `zero_module` util
1. Add tests for each component

Reviewed By: pbontrager

Differential Revision: D47709486

fbshipit-source-id: 16da8eaeac65c64b40b5afe266a12540d95a7bcc
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D47709486

@facebook-github-bot
Copy link
Contributor

This pull request has been merged in 97264a2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. fb-exported Merged
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@abhinavarora @facebook-github-bot @codecov-commenter