An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence

Qizhen Zhang, Ankush Garg, Jakob Nicolaus Foerster, Niladri S. Chatterji, Kshitiz Malik, Mike Lewis

Keywords: LLM pretraining, training dynamics, data quality

TL;DR: We present a systematic empirical study of whether noisy data causes LLM pretraining divergences and how it does so.

Abstract:

Large-scale pretraining datasets drive the success of large language models (LLMs). However, these web-scale corpora inevitably contain large amounts of noisy data due to unregulated web content, or randomness inherent in data. Although LLM pretrainers often speculate that such noise contributes to instabilities in large-scale LLM pretraining and, in the worst cases, loss divergence, this phenomenon remains poorly understood.
In this work, we present a systematic empirical study of whether noisy data causes LLM pretraining divergences and how it does so. By injecting controlled synthetic uniformly random noise into otherwise clean datasets, we analyze training dynamics across model sizes ranging from 480M to 5.2B parameters.
We show that noisy data indeed induces training loss divergence, and that the probability of divergence depends strongly on the noise type, amount of noise, and model scale. We further find that noise-induced divergences exhibit activation patterns distinct from those caused by high learning rates \citep{wortsman2023small}, and we provide diagnostics that differentiate these two failure modes. Together, these results provide one of the first large-scale, controlled characterizations of how noisy data affects loss divergence in LLM pretraining.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 19