Series of Shell scripts for Machine Translation, to serve as examples for MT projects. These scripts are derived from scripts I used during for my masters thesis (M.Sc. in Computer Science).
Note: In all these examples, "src" is the source language and "tgt" is the target language, to be replaced for example with the actual ISO codes for the chosen languages.
Scripts to train a Transformer NMT model, based on train, valid and test sets. These scripts are partly adapted from fairseq scripts (https://github.com/facebookresearch/fairseq)
Pre-processing (tokenizing, punctuation standardization, etc.) and segmentation steps. Segmentation is performed by training & applying a sentencepiece BPE model. Preprocessing requires Moses scripts and segmentation requires fairseq scripts.
Training and evaluation of Transformer model using pre-processed train, valid and test sets. Both steps require fairseq scripts.
Interactively use the model trained with train-evaluate_transformer.sh script to produce translations (from src to tgt). This requires fairseq scripts.
Scripts to train a SMT model, based on train, valid and test sets. These scripts are partly adapted from Moses SMT examples (http://www2.statmt.org/moses/) and require Moses scripts to function properly.
Pre-process all sets (tokenizing, truecasing).
Train SMT model.
Tune model resulting from training step on valid parallel set.
Produces translation hypotheses for sentences in test set, compare them to reference translation and produce BLEU (sacrebleu) and Chrf++ scores. The scoring step requires sacrebleu (https://github.com/mjpost/sacrebleu).