Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Script for comparison of CoCa zero-shot to open_clip implementation #507

Draft
wants to merge 5 commits into
base: gh/ebsmothers/23/base
Choose a base branch
from

Conversation

ebsmothers
Copy link
Contributor

@ebsmothers ebsmothers commented Nov 1, 2023

@ebsmothers ebsmothers mentioned this pull request Nov 1, 2023
ebsmothers added a commit that referenced this pull request Nov 1, 2023
ghstack-source-id: ab00a880c06992df61427d3fadaf38050bda0995
Pull Request resolved: #507
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 1, 2023
@ebsmothers ebsmothers marked this pull request as draft November 1, 2023 16:43
ebsmothers added a commit that referenced this pull request Nov 2, 2023
ghstack-source-id: 7313eca3f5c6168073e94bd8922f348eb0994afd
Pull Request resolved: #507
@ebsmothers
Copy link
Contributor Author

ebsmothers commented Nov 2, 2023

Note: This PR is not to land and is instead meant for verification purposes.

Some modifications to CoCa class and an evaluation script to validate that our implementation matches the ImageNet zero-shot performance of the one in open_clip repo on their pretrained ViT-L/14.

Test:

python eval.py --pretrained /home/ebs/checkpoints/open_clip_coca_vit_l_14_laion2B-s13B-b90k-remapped.pt
...
TorchMultimodal zero-shot accuracy: {'imagenet-zeroshot-test-top1': 0.75628, 'imagenet-zeroshot-test-top5': 0.94284}
...
    accuracy                          0.757     50000
   macro avg      0.768     0.757     0.751     50000
weighted avg      0.768     0.757     0.751     50000

ebsmothers added a commit that referenced this pull request Nov 2, 2023
ghstack-source-id: d89ac55d9ec0d26e0981b7e2fa94ea85de046088
Pull Request resolved: #507
ebsmothers added a commit that referenced this pull request Nov 2, 2023
ghstack-source-id: f004f4797350fd79c37437a2f87fa8fa32f48068
Pull Request resolved: #507
ebsmothers added a commit that referenced this pull request Nov 15, 2023
ghstack-source-id: ab570aec8104d8a09a9bf637a5c434bf479d8131
Pull Request resolved: #507
@ebsmothers
Copy link
Contributor Author

@ebsmothers has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants