The scripts should be pretty self-explanatory, just adjust the values in the scripts to your desired values and train on any .txt file!
-
Install
pytorch(gpu recommended) -
pip install -r requirements.txt
Note: Does it work? Yes, but it just kind of spits out random words and sentences. But hey, its an llm from scratch!