Skip to content

Latest commit

 

History

History
25 lines (17 loc) · 2.07 KB

README.md

File metadata and controls

25 lines (17 loc) · 2.07 KB

ARPA: Armenian Paraphrase Detection Corpus and Models

We provide sentential paraphrase detection train, test datasets as well as BERT-based models for the Armenian language.

To sentences in the dataset are taken from Hetq and Panarmenian news articles.

To generate paraphrase for the sentences, we used back translation from Armenian to English. We repeated the step twice, after which the generated paraphrases were manually reviewed. Invalid sentences were filtered out, while the rest were labelled as either paraphrase, near paraphrase or non-paraphrase (with 1, 0, -1 labels respectively). The sentences are divided into train and test sets. Test examples were reviewed by 3 different annotators. In addition, to increase the number of non-paraphrase pairs, we padded the dataset with automatically generated negative examples, including pairs of consecutive sentences and random pairs.

Number of examples Total Paraphrase Non-paraphrase (near paraphrase)
Train 4233 1339 2684 (210)
Test 1682 1021 449 (212)

We finetuned Multilingual BERT on several training sets, including the proposed ARPA dataset, and evaluated their performance on our test set. During training and evaluation, near paraphrase and non-paraphrase pairs were combined into one class. The results are provided below:

BERT Model Train set F1 Acc.
Multilingual BERT ARPA train set 84.27 78.06
Multilingual BERT Paraphraser.ru train set machine-translated into Armenian 83.81 77.09
Multilingual BERT MRPC train set machine-translated into Armenian 80.07 69.87
Multilingual BERT All of the above combined 84 77.6

The model trained on ARPA is available for use, and can be downloaded using this link.

For more details about the models and dataset construction, refer to the paper.