Script for comparison of CoCa zero-shot to open_clip implementation

[ebsmothers]

Stack from ghstack (oldest at bottom):

• -> Script for comparison of CoCa zero-shot to open_clip implementation #507
• Add CoCa model #506

Differential Revision: D51332626

[ebsmothers]

Note: This PR is not to land and is instead meant for verification purposes.

Some modifications to CoCa class and an evaluation script to validate that our implementation matches the ImageNet zero-shot performance of the one in open_clip repo on their pretrained ViT-L/14.

Test:

python eval.py --pretrained /home/ebs/checkpoints/open_clip_coca_vit_l_14_laion2B-s13B-b90k-remapped.pt
...
TorchMultimodal zero-shot accuracy: {'imagenet-zeroshot-test-top1': 0.75628, 'imagenet-zeroshot-test-top5': 0.94284}
...
    accuracy                          0.757     50000
   macro avg      0.768     0.757     0.751     50000
weighted avg      0.768     0.757     0.751     50000

[ebsmothers]

@ebsmothers has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.