This work provides extensive empirical results on training LMs to count. We find that while traditional RNNs trivially achieve inductive counting, Transformers have to rely on positional embeddings to count out-of-domain. Modern RNNs (e.g. rwkv, mamba) also largely underperform traditional RNNs in generalizing counting inductively.

Language Model project is a Java-based language and N-Gram model. It predicts up to two words based on a single word input and provides detailed text analysis statistics. Demonstrating advanced object-oriented programming and design principles, it is a valuable tool for predictive text input and linguistic analysis.