PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs

Oskar van der Wal, Pietro Lesci, Max Müller-Eberstein, Naomi Saphra, Hailey Schoelkopf, Willem Zuidema, Stella Biderman

Keywords: language models, training dynamics, interpretability, memorization, robustness, training stability

TL;DR: We introduce a set of 45 new training runs for the Pythia models suite to study stability across size and random seed.

Abstract:

Understanding the stability of language model pre-training and its effects on downstream performance is still understudied.
Prior work shows that the training process can yield significantly different results in response to slight variations in initial conditions, e.g., the random seed.
Crucially, resources to study pre-training stability in language models are still lacking, especially for decoder-only models.
We introduce the PolyPythias, a set of 45 new training runs for the Pythia model suite: 9 new seeds across 5 model sizes, from 14M to 410M parameters, resulting in about 7k new checkpoints that we release.
Using these new 45 training runs, in addition to the 5 already available, we study the effects of different initial conditions determined by the seed---i.e., parameters' initialisation and data order---on (i) downstream performance, (ii) learned linguistic representations, and (iii) emergence of training phases.
In addition to common scaling behaviours, our analyses generally reveal highly consistent training dynamics across both model sizes and initial conditions.
Additionally, the new seeds for each model allow us to identify outlier training runs and delineate their characteristics.
Our findings show the potential of using these methods to predict training stability.

Primary Area: interpretability and explainable AI

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 11899