AttributeError: 'DistilBertConfig' object has no attribute 'intermediate_size'

[jknikita]

Hello
I was trying to use the Distilbert model but it's giving me this error (AttributeError: 'DistilBertConfig' object has no attribute 'intermediate_size') Can you please help me with this?
Thank you

[iliaschalkidis]

Hi @jknikita, could you point to a specific point in the codebase and/or provide a more detailed error message...?

[iliaschalkidis]

It seems that DistilBERT has its very own configuration and several standard fields are missing or are named differently.

I would propose you to consider the similarly sized distilroberta-base that uses a standard roberta configuration file.

[jknikita]

I did use distilroberta-base but got this error

Can you help me with this also? Thank you.

[iliaschalkidis]

Which version of torch are you using?

[jknikita]

1.12.1+cu113

[iliaschalkidis]

The distilroberta-base is a standard RoBERTa model, and it's loaded properly, thus your code trains for an epoch and then stops for evaluation and crashes, which is very weird. I suspect either an issue with the torch version or maybe fp16 support.

Could you please downgrade to 1.11.0 and possibly 1.9.0, if 1.11.0 still has issues?

# UNINSTALL TORCH 1.12
pip uninstall torch
# TORCH 1.11 + CUDA 11.3
pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113
# TORCH 1.9 + CUDA 11.1
pip install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html

Does roberta-base work properly?