Efficient Softmax Approximation

Implementations of Blackout and Adaptive Softmax for efficiently calculating word distribution for language modeling of very large vocabularies.

LSTM language models are derived from rnnlm_chainer.

Available output layers are as follows

Linear + softmax with cross entropy loss. A usual output layer.
--share-embedding: A variant using the word embedding matrix shared with the input layer for the output layer.
--adaptive-softmax: Adaptive softmax
--blackout: BlackOut (BlackOut is not faster on GPU.)

Efficient softmax approximation for GPUs
Edouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, Hervé Jégou, ICML 2017
paper
authors' Lua code

BlackOut: Speeding up Recurrent Neural Network Language Models With Very Large Vocabularies
Shihao Ji, S. V. N. Vishwanathan, Nadathur Satish, Michael J. Anderson, Pradeep Dubey, ICLR 2016
paper
authors' C++ code

How to Run

python -u train.py -g 0

For wikitext, run prepare_wikitext.sh for downloading the datasets.