DiLoCo: Distributed Low-Communication Training of Language Models

MarcAurelio Ranzato

Abstract:

Large language models (LLM) have become a critical component in many applications of machine
learning. However, standard approaches to training LLM require a large number of tightly interconnected
accelerators, with devices exchanging gradients and other intermediate states at each optimization
step. While it is difficult to build and maintain a single computing cluster hosting many accelerators, it
might be easier to find several computing clusters each hosting a smaller number of devices. In this
work, we propose a distributed optimization algorithm, Distributed Low-Communication (DiLoCo), that
enables training of language models on islands of devices that are poorly connected. The approach
is a variant of federated averaging, where the number of inner steps is large, the inner optimizer is
AdamW, and the outer optimizer is Nesterov momentum. On the widely used C4 dataset, we show that
DiLoCo on 8 workers performs as well as fully synchronous optimization while communicating 500
times less. DiLoCo exhibits great robustness to the data distribution of each worker. It is also robust
to resources becoming unavailable over time, and vice versa, it can seamlessly leverage resources that
become available during training.