Adding MT5 support

[sotwi]

Pull request to address #568.

I followed the updated guide for Adapters to a Model and did a very quick port. I followed the approach that the mBART implementation took(they reused the BART mixins, I reused the T5 mixins) so the changes were minimal.

I hope it works.

[sotwi]

There appears to be an issues when loading the public mt5 weights into the AdapterModel.
I have no idea what it is, but I think it has to do with those weights already including a lm_head.weight layer in their state dict. That seems to cause issues when initializing the Flexible Heads.

[sotwi]

When I try to load public mt5 weights (say mt5-small) with either AutoAdapterModel or MT5AdapterModel I get the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/wsotomar/anaconda3/envs/dev_adapters/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py", line 566, in from_pretrained
    return model_class.from_pretrained(
  File "/home/wsotomar/anaconda3/envs/dev_adapters/lib/python3.8/site-packages/transformers/modeling_utils.py", line 3480, in from_pretrained
    ) = cls._load_pretrained_model(
  File "/home/wsotomar/Code/adapters/src/adapters/heads/base.py", line 969, in _load_pretrained_model
    return super()._load_pretrained_model(
  File "/home/wsotomar/anaconda3/envs/dev_adapters/lib/python3.8/site-packages/transformers/modeling_utils.py", line 3752, in _load_pretrained_model
    raise ValueError(
ValueError: The state dictionary of the model you are trying to load is corrupted. Are you sure it was properly saved?

I then tried loading the weights with the transformers MT5Model class, which gets rid of the lm_head layer that comes prepackaged with the weights. I saved that model locally and then loaded it with both AutoAdapterModel and MT5AdapterModel without any issues.

I really think that the inclusion of the lm_head on the original weights is messing with the loading process of ModelWithFlexibleHeadsAdaptersMixin and that's why its failing. But I can't quite find the exact issue nor how to fix it.

If anyone wants to have a look at it that would be great.

[calpt]

@sotwi thanks so much for your work on this so far! will look into the issue you mentioned shortly

[calpt]

Thanks again for working on this. I've looked into the issue and am working fixing this separately in #640 (for both the failing tests and the lm_head error). Once the fix there is ready, this PR is good to merge from my side!

[sotwi]

Thank you for your help @calpt!!!
I am glad it is working now!