Fairseq(-py) is a sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling and other text generation tasks. This repo is forked from Fairseq and includes changes to run models on Intel® Gaudi® AI accelerators.

We provide reference implementations of various sequence modeling papers:

List of implemented papers

• Convolutional Neural Networks (CNN)
  • Language Modeling with Gated Convolutional Networks (Dauphin et al., 2017)
  • Convolutional Sequence to Sequence Learning (Gehring et al., 2017)
  • Classical Structured Prediction Losses for Sequence to Sequence Learning (Edunov et al., 2018)
  • Hierarchical Neural Story Generation (Fan et al., 2018)
  • wav2vec: Unsupervised Pre-training for Speech Recognition (Schneider et al., 2019)
• LightConv and DynamicConv models
  • Pay Less Attention with Lightweight and Dynamic Convolutions (Wu et al., 2019)
• Long Short-Term Memory (LSTM) networks
  • Effective Approaches to Attention-based Neural Machine Translation (Luong et al., 2015)
• Transformer (self-attention) networks
  • Attention Is All You Need (Vaswani et al., 2017)
  • Scaling Neural Machine Translation (Ott et al., 2018)
  • Understanding Back-Translation at Scale (Edunov et al., 2018)
  • Adaptive Input Representations for Neural Language Modeling (Baevski and Auli, 2018)
  • Lexically constrained decoding with dynamic beam allocation (Post & Vilar, 2018)
  • Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context (Dai et al., 2019)
  • Adaptive Attention Span in Transformers (Sukhbaatar et al., 2019)
  • Mixture Models for Diverse Machine Translation: Tricks of the Trade (Shen et al., 2019)
  • RoBERTa: A Robustly Optimized BERT Pretraining Approach (Liu et al., 2019)
  • Facebook FAIR's WMT19 News Translation Task Submission (Ng et al., 2019)
  • Jointly Learning to Align and Translate with Transformer Models (Garg et al., 2019)
  • Multilingual Denoising Pre-training for Neural Machine Translation (Liu et at., 2020)
  • Neural Machine Translation with Byte-Level Subwords (Wang et al., 2020)
  • Unsupervised Quality Estimation for Neural Machine Translation (Fomicheva et al., 2020)
  • wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (Baevski et al., 2020)
  • Generating Medical Reports from Patient-Doctor Conversations Using Sequence-to-Sequence Models (Enarvi et al., 2020)
  • Linformer: Self-Attention with Linear Complexity (Wang et al., 2020)
  • Cross-lingual Retrieval for Iterative Self-Supervised Training (Tran et al., 2020)
  • Deep Transformers with Latent Depth (Li et al., 2020)
  • Unsupervised Cross-lingual Representation Learning for Speech Recognition (Conneau et al., 2020)
  • Self-training and Pre-training are Complementary for Speech Recognition (Xu et al., 2020)
  • Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training (Hsu, et al., 2021)
  • Unsupervised Speech Recognition (Baevski, et al., 2021)
  • Simple and Effective Zero-shot Cross-lingual Phoneme Recognition (Xu et al., 2021)
  • VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (Xu et. al., 2021)
  • VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding (Xu et. al., 2021)
  • NormFormer: Improved Transformer Pretraining with Extra Normalization (Shleifer et. al, 2021)
• Non-autoregressive Transformers
  • Non-Autoregressive Neural Machine Translation (Gu et al., 2017)
  • Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement (Lee et al. 2018)
  • Insertion Transformer: Flexible Sequence Generation via Insertion Operations (Stern et al. 2019)
  • Mask-Predict: Parallel Decoding of Conditional Masked Language Models (Ghazvininejad et al., 2019)
  • Levenshtein Transformer (Gu et al., 2019)
• Finetuning
  • Better Fine-Tuning by Reducing Representational Collapse (Aghajanyan et al. 2020)

Requirements and Installation

• Please follow the instructions provided in the Gaudi Installation Guide to set up the environment. To achieve the best performance, please follow the methods outlined in the Optimizing Training Platform guide. The guides will walk you through the process of setting up your system to run the model on Gaudi.
• To install fairseq and develop locally:

git clone https://github.com/HabanaAI/fairseq
cd fairseq
pip install --editable ./

Getting Started

List of models for which training has been tested on Gaudi devices:

• Transformer

In order to train another model available in fairseq (other than those listed above) on Gaudi device, please follow the instructions below,

• Use "--hpu" argument when invoking command-line tools such as fairseq-train, fairseq-interactive, fairseq-generate etc.
• Enable mixed precision training by using "--hpu-mixed-precision-mode autocast" when invoking command-line tools such as fairseq-train.

License

fairseq(-py) is MIT-licensed. The license applies to the pre-trained models as well.

Citation

Please cite as:

@inproceedings{ott2019fairseq,
  title = {fairseq: A Fast, Extensible Toolkit for Sequence Modeling},
  author = {Myle Ott and Sergey Edunov and Alexei Baevski and Angela Fan and Sam Gross and Nathan Ng and David Grangier and Michael Auli},
  booktitle = {Proceedings of NAACL-HLT 2019: Demonstrations},
  year = {2019},
}