UMBCLU at SemEval-2024 Task 1A and 1C: Semantic Textual Relatedness with and without machine translation

This is the official implementation of our final submission on SemEval 2024, Task 1A. Paper is available on arXiv.

Run Locally

Clone

  git clone https://github.com/dipta007/SemEval24-task1

Go to the project directory

  cd SemEval24-task1

Install dependencies

  conda env create -f environment.yml 
  conda activate sem24_task1

Download Data

 gdown https://drive.google.com/drive/folders/1AfwbgZVGy6svWLqkiiejvq4dQMBLU9a4 -O ./data --folder

Run trainer

  python src/train.py --exp_name=EXP_NAME

Final Model Hyperparameters

 'accumulate_grad_batches': 32,
 'batch_size': 16,
 'data_dir': './data/Track A',
 'early_stopping_patience': 10,
 'enc_dropout': 0.1,
 'enc_pooling': 'mean',
 'lr': 1e-05,
 'max_epochs': -1,
 'model_name': 'sentence-transformers/all-distilroberta-v1',
 'monitoring_metric': 'valid/corr',
 'monitoring_mode': 'max',
 'seed': 42,
 'validate_every': 1.0,
 'weight_decay': 0.01

Test Result

Language	Spearman Correlation
English	0.8124657642704284
Hausa	0.6402557259721893
Kinyarwanda	0.6806672639024565
Marathi	0.8406501120112206
Moroccan Arabic	0.7447707874574931
Spanish	0.6382694075818184
Telugu	0.8255452084837941