Character-Aware Neural Language Models

Code for the paper Character-Aware Neural Language Models (AAAI 2016).

A neural language model (NLM) built on character inputs only. Predictions are still made at the word-level. The model employs a convolutional neural network (CNN) over characters to use as inputs into an long short-term memory (LSTM) recurrent neural network language model (RNN-LM). Also optionally passes the output from the CNN through a Highway Network, which improves performance.

Much of the base code is from Andrej Karpathy's excellent character RNN implementation.

Requirements

Code is written in Lua and requires Torch. It also requires the nngraph and the luautf8 packages, which can be installed via:

luarocks install nngraph
luarocks install luautf8

GPU usage will additionally require cutorch and cunn packages:

luarocks install cutorch
luarocks install cunn

cudnn will result in a good (8x-10x) speed-up for convolutions, so it is highly recommended. This will make the training time of a character-level model be somewhat competitive against a word-level model (1500 tokens/sec vs 3000 tokens/sec for the large character/word-level models described below).

git clone https://github.com/soumith/cudnn.torch.git
cd cudnn.torch
luarocks make cudnn-scm-1.rockspec

Data

Data should be put into the data/ directory, split into train.txt, valid.txt, and test.txt

Each line of the .txt file should be a sentence. The English Penn Treebank (PTB) data (Tomas Mikolov's pre-processed version with vocab size equal to 10K, widely used by the language modeling community) is given as the default.

The paper also runs the models on non-English data (Czech, French, German, Russian, and Spanish), from the ICML 2014 paper Compositional Morphology for Word Representations and Language Modelling by Jan Botha and Phil Blunsom. This can be downloaded from Jan's website.

For ease of use, we provide a script to download the non-English data (get_data.sh). The script also saves the downloaded data into the relevant folders.

Note on PTB

The PTB data above does not have end-of-sentence tokens for each sentence, and hence these must be manually appended. This can be done by adding -EOS '+' to the script (obviously you can use other characters than + to represent an end-of-sentence token---we recommend a single unused character).

The non-English data already have end-of-sentence tokens for each line so, you want to add -EOS '' to the command line.

Unicode in Lua

Lua is unicode-agnostic (each string is just a sequence of bytes) so we use the luautf8 package to deal with languages where a character can be more than one byte (e.g. Russian). Many thanks to vseledkin for alerting us to the fact that previous version of the code did not take this account!

Model

Here are some example scripts. Add -gpuid 0 to each line to use a GPU (which is required to get any reasonable speed with the CNN), and -cudnn 1 to use the cudnn package. Scripts to reproduce the results of the paper can be found under run_models.sh

Character-level models

Large character-level model (LSTM-CharCNN-Large in the paper). This is the default: should get ~82 on valid and ~79 on test. Takes ~5 hours with cudnn.

th main.lua -savefile char-large -EOS '+'

Small character-level model (LSTM-CharCNN-Small in the paper). This should get ~96 on valid and ~93 on test. Takes ~2 hours with cudnn.

th main.lua -savefile char-small -rnn_size 300 -highway_layers 1 
-kernels '{1,2,3,4,5,6}' -feature_maps '{25,50,75,100,125,150}' -EOS '+'

Word-level models

Large word-level model (LSTM-Word-Large in the paper). This should get ~89 on valid and ~85 on test.

th main.lua -savefile word-large -word_vec_size 650 -highway_layers 0 
-use_chars 0 -use_words 1 -EOS '+'

Small word-level model (LSTM-Word-Small in the paper). This should get ~101 on valid and ~98 on test.

th main.lua -savefile word-small -word_vec_size 200 -highway_layers 0 
-use_chars 0 -use_words 1 -rnn_size 200 -EOS '+'

Combining both

Note that if -use_chars and -use_words are both set to 1, the model will concatenate the output from the CNN with the word embedding. We've found this model to underperform a purely character-level model, though.

Evaluation

By default main.lua will evaluate the model on test data after training, but this will use the last epoch's model, and also will be slow due to the way the data is set up.

Evaluation on test can be performed via the following script:

th evaluate.lua -model model_file.t7 -data_dir data/ptb -savefile model_results.t7

Where model_file.t7 is the path to the best performing (on validation) model. This will also save some basic statistics (e.g. perplexity by token) in model_results.t7.

Hierarchical Softmax

Training on a larger vocabulary (e.g. 100K+) will require hierarchical softmax (HSM) to train at a reasonable speed. You can use the -hsm option to do this. For example -hsm 500 will randomly split the vocabulary into 500 clusters of (approximately) equal size. -hsm 0 is the default and will not use HSM. -hsm -1 will automatically choose the number of clusters for you, by choosing the integer closest to sqrt(|V|).

Batch Size

If training on bigger datasets you should probably use a larger batch size (e.g. -batch_size 100).

Licence

MIT