Scalable Training of Language Models using JAX pjit and TPUv4

Anonymous

Keywords: Distributed Training, Training Infrastructure, Model Parallelism, Training Efficiency, Efficient Methods for NLP

TL;DR: We describe our framework for training large language models and its utilization of the state-of-the-art hardware and software to perform efficient, large-scale, parallel computation.

Abstract: Modern large language models require distributed training strategies due to their size. The challenges of efficiently and robustly training them are met with rapid developments on both software and hardware frontiers. In this technical report, we explore challenges and design decisions associated with developing a scalable training framework, and present a quantitative analysis of efficiency improvements coming from adopting new software and hardware solutions.