This is the repository for DisTrO (Distributed Training Over-The-Internet), a family of low latency distributed optimizers that reduce inter-GPU communication requirements by three to four orders of magnitude.
- Aug. 26th, 2024: Preliminary Report
- Coming Soon: Paper and Code
- In The Near Future: 👀
Join us on Discord if you're interested in helping research and build the future of distributed training.