A Mechanistic Analysis of Low-Precision Instabilities in Microscaling Formats

Huangyuan Su, Mujin Kwun, Stephanie Gil, Sham M. Kakade, Nikhil Anand

Keywords: large language models, precision, quantization, microscaling, scaling law, pretraining

TL;DR: We identify the mechanisms of precision-driven instabilities in LLM training and potential mitigation strategies.

Abstract: Training large language models is expensive and compute-bound, and it must be repeated as models scale, algorithms improve, and new data is collected. To address this, next-generation hardware accelerators like NVIDIA’s Blackwell increasingly support lower-precision arithmetic formats, including Microscaling (MX) formats. In this work, we investigate the challenges and viability of block-scaled precision formats during model training. Across a broad sweep of weight-activation precision combinations and compute budgets from \( 2 \times 10^{17} \) to \( 4.8 \times 10^{19} \) FLOPs, we generally observe that training in MX formats exhibits sharp, stochastic instabilities in the loss, particularly at larger compute scales. To explain this phenomenon, we conduct controlled experiments and ablations on a smaller proxy model that exhibits instability behavior similar to the language model, sweeping across architectural settings, hyperparameters, and precision formats. These experiments motivate a simple model in which multiplicative gradient bias introduced by the quantization of layer-norm affine parameters and a small fraction of activations can trigger runaway divergence. Through \textit{in situ} intervention experiments on our proxy model, we demonstrate that instabilities can be averted or delayed by modifying precision schemes mid-training. Guided by these findings, we evaluate stabilization strategies in the LLM setting and show that certain hybrid configurations recover performance competitive with full-precision training.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 1136