Lectures on Computational Linguistics 2021

Tranining a sentence classifier using BERT-based models (in Pytorch and compatible with HuggingFace)

Author: Danilo Croce

This is a Pytorch (+ Huggingface transformers) implementation of a "simple" text classifier defined using BERT-based models. In this lab we will see how it is simple to use BERT for a sentence classification task, obtaining state-of-the-art results in few lines of python code.

In a nutshell, we will "implement" this architecture:

Given an input sentence, we will use a BERT-based architecture to encode it and a linear classifier is used to associate the produced vector to one of the c classes considered in the classification task.

Most importantly, we will show how to use the same code but using different transformer based architectures, such as:

• RoBERTa or Electra in English;
• multilingual BERT or XLMRoBERTA;
• specific encoders trained on Italian corpora.

Moreover, we will evaluate our code in two sentence classification tasks:

• Question Classification in English
• Subjectivity identification in Italian

by (hopefully) changing only one or two lines of code.