pipeline-steps.md

layout	title	nav_order	has_children
default	Pipeline steps	3	true

Pipeline steps

The pipeline steps are based on the train-student recipe. They can be represented as a Directly Acyclic Graph (DAG) (svg).

This scheme is slightly outdated, since it was generated using Snakemake and we stopped maintaining it, but it shows the idea of dependencies.

Toolchain

Installs dependencies and compiles Marian and other tools.

Data downloading

Downloads datasets and samples an appropriate amount from the sentences. The time depends on dataset size. The sampling of huge mono datasets (100M+ sentences) is the most intensive operation.

Uses datasets.train configuration section.

Analyze data

Runs data analysis on the downloaded datasets and outputs charts. For example a distribution of sentence length in a dataset.

Data cleaning

Basic preprocessing, dataset specific, language specific, rule based and other attempts to clean noisy data in parallel and monolingual datasets. Good parallelization across CPU cores. To make cleaning of a new language more efficient add it to clean_parallel.py.

Alternatively use OpusCleaner.

Bicleaner AI

Filters noisy sentence pairs in a parallel corpus using bicleaner-ai classifier. Cleaning thresholds are configurable per dataset, see [Dataset cleaning](##Dataset cleaning)

See more details on the Bicleaner page.

Merge and dedupe

Merges clean datasets into one and applies deduplication.

Training vocabulary

Trains SentencePiece vocabulary/tokenizer model on parallel corpus.

See more details on choosing the size of vocabulary here.

Training backward model

Trains a shallow sequence to sequence RNN model in an opposite direction. It is useful for back-translations and cross entropy filtering. It is based on a marian example.

Augmentation with back-translations

Translates monolingual corpus combined from monolingual datasets in target language using the backward model.

It is more useful for low-resource languages but is still recommended for high-resource ones as well.

Training alignments

Trains whitespace-tokenized alignments accepted by OpusTrainer using eflomal library.

It trains alignments separately for origninal parallel, backtranslated and student corpus. Backtranslated and student steps use eflobal priors extracted from the alignments trained for the original parallel corpus. It can improve accuracy for a smaller corpus as well as performance.

It works with uncompressed datasets, so it can be heavy on disk.

Training teacher

Trains several big transformer models on the augmented dataset. They will be later used for decoding as an ensemble of models.

Translation by teachers

Translates the corpus and the monolingual data in source language (configurable in datasets.mono-src) using the ensemble of teacher models.

This is the heaviest part of the pipeline but highly parallelizable.

Cross-entropy filtering

Scores the translated corpus with the backward model and removes a part of the corpus with the lowest scores to reduce noise.

At this point we work with huge datasets that can be very disk intensive.

Training shortlist

Trains SentencePiece tokenized alignments using elfomal similar to the alignments steps and then extracts lexical shortlist using extract_lex tool.

Some tools require uncompressed datasets on disk, and they are huge at this point. Good CPU parallelization.

Training student

Trains a small transformer student model on the filtered data and using the alignments. OpusTrainer remaps the alignment to SentencePiece. See more details on the OpusTrainer page.

Fine-tuning student

Fine-tunes the student model by emulating 8bit GEMM during training. Converges very quickly and then degrades.

Quantization

Applies 8 bit quantization to the fined-tuned student model. Marian CPU threads must be set to 1 for this step.

Evaluation

Calculates metrics for all models (BLEU, chrF) using SacreBLEU. It runs Marian decoding GPU for all models except the quantized ones that it runs on CPU.

It uses datasets.test configuration section.

Export

Exports the trained model and the shortlist to (bergamot-translator)(https://github.com/mozilla/bergamot-translator) format.

Resource usage

Step	Bottleneck
Installation	CPU
Data downloading	Network, Disk
Analyze data	CPU, Disk
Data cleaning	CPU
Bicleaner	GPU
Merge and dedupe	CPU, Disk
Training vocabulary	CPU
Training backward model	GPU
Augmentation with back-translations	GPU
Training alignments	CPU, Disk
Training teacher	GPU
Translation by teacher	GPU
Cross-entropy filtering	GPU, CPU, Disk
Training shortlist	CPU, Disk
Training student	GPU
Fine-tuning student	GPU
Quantizaiton	CPU
Evaluation	GPU
Export