Load Timm ViT model weights to huggingface ViT implementation

[anjali-chadha]

Hello folks!

I have a model which was originally trained using timm's VisionTransformer implementation.
I am interested in loading this model weights into VisionTransformer implementation by huggingface.

I have a few questions:

1. How can we achieve this model conversion?

   I would like to know the steps or code modifications required to load the model weights from timm's VisionTransformer implementation into the huggingface's native VisionTransformer implementation.

2. What are the differences between the two implementations?

   Understanding the differences between the timm and huggingface implementations would help me assess the impact of using one or the other in my team's application.

3. Any best practices or considerations while performing this conversion?

Thank you in advance for your help!

Related discussion thread, but didn't answer my questions fully: huggingface/transformers#19305

[rwightman]

@anjali-chadha there is a conversion script in transformers, not sure it's up to date with the latest but should be minimal changes to make it work if out of date https://github.com/huggingface/transformers/blob/main/src/transformers/models/vit/convert_vit_timm_to_pytorch.py

Not all timm vit models will have equivalent arch in transformers, there are some variations would need code additions on the transformer side. Most of the main ones would work though. Beware that norm epsilon tends to be wrong in many transformers vit model configs, it's 1e-6 not 1e-12.