README.md

nanoGPT code-base for self-trained language models

Link to the paper: Collapse of Self-trained Langauge Models

install

pip install torch numpy transformers datasets tiktoken wandb tqdm

To run the code do:

python train.py config/finetune_collapse.py