The purpose of this project is to explore the use of LSTM models for the application of identifying code switching using the PyTorch library. The models generated by this project can be used take sentences given as an input, and make predictions for the language used for each individual word in the sentence.
The project implements two types of LSTM models, a single-layer LSTM model, which generates word embeddings from a given word vector, performs an LSTM cycle, and generates predictions for tags for each word in the sentence, and a double-layer LSTM model, which performs and LSTM cycle on the characters of each given word, and concatenates the final hidden state to the word embedding, and the 2nd LSTM process is performed using this concatenated tensor.
The results for different types of models can be test using the testModels function, which calculates and displays the accuracy rate for each generated model. Here is the output for the sample models included in this repo:
Testing singleLSTM...
LSTMTagger(
(word_embeddings): Embedding(12221, 32)
(lstm): LSTM(32, 32, bidirectional=True)
(hidden2tag): Linear(in_features=64, out_features=3, bias=True)
)
Final stats: Trials 9583, Successes 8157, Success Rate 0.8511948241677971
Testing singleLSTM_uni...
LSTMTagger(
(word_embeddings): Embedding(12221, 32)
(lstm): LSTM(32, 32)
(hidden2tag): Linear(in_features=32, out_features=3, bias=True)
)
Final stats: Trials 9583, Successes 7827, Success Rate 0.8167588437858708
Testing singleLSTM01 - 2Layer...
LSTMTagger(
(word_embeddings): Embedding(12221, 32)
(lstm): LSTM(32, 32, num_layers=2, bidirectional=True)
(hidden2tag): Linear(in_features=64, out_features=3, bias=True)
)
Final stats: Trials 9583, Successes 8074, Success Rate 0.8425336533444642
Testing DoubleLSTM - bidir - 2layer...
LSTMTagger2Layer(
(word_embeddings): Embedding(12221, 32)
(char_embeddings): Embedding(288, 16)
(lstm_words): LSTM(64, 32, num_layers=2, bidirectional=True)
(lstm_char): LSTM(16, 16, num_layers=2, bidirectional=True)
(hidden2tag): Linear(in_features=64, out_features=3, bias=True)
)
Final stats: Trials 9583, Successes 8477, Success Rate 0.8845872899926954
Testing DoubleLSTM - uni...
LSTMTagger2Layer(
(word_embeddings): Embedding(12221, 32)
(char_embeddings): Embedding(288, 16)
(lstm_words): LSTM(48, 32)
(lstm_char): LSTM(16, 16)
(hidden2tag): Linear(in_features=32, out_features=3, bias=True)
)
Final stats: Trials 9583, Successes 8718, Success Rate 0.9097359908170719
Testing DoubleLSTM...
LSTMTagger2Layer(
(word_embeddings): Embedding(12221, 32)
(char_embeddings): Embedding(288, 16)
(lstm_words): LSTM(64, 32, bidirectional=True)
(lstm_char): LSTM(16, 16, bidirectional=True)
(hidden2tag): Linear(in_features=64, out_features=3, bias=True)
)
Final stats: Trials 9583, Successes 8687, Success Rate 0.9065010956902849
Testing DoubleLSTMepoch1...
LSTMTagger2Layer(
(word_embeddings): Embedding(12221, 32)
(char_embeddings): Embedding(288, 16)
(lstm_words): LSTM(64, 32, bidirectional=True)
(lstm_char): LSTM(16, 16, bidirectional=True)
(hidden2tag): Linear(in_features=64, out_features=3, bias=True)
)
Final stats: Trials 9583, Successes 7644, Success Rate 0.7976625273922571
Testing DoubleLSTMepoch3...
LSTMTagger2Layer(
(word_embeddings): Embedding(12221, 32)
(char_embeddings): Embedding(288, 16)
(lstm_words): LSTM(64, 32, bidirectional=True)
(lstm_char): LSTM(16, 16, bidirectional=True)
(hidden2tag): Linear(in_features=64, out_features=3, bias=True)
)
Final stats: Trials 9583, Successes 8485, Success Rate 0.8854221016383178
The results show the learning rate of the model through each epoch, the effectiveness of bidirection LSTMs vs monodirectional, and the performance improvement gained from the double LSTM process vs the single LSTM process.
Please enjoy responsibly.