Loss in the Crowd: Hidden Breakthroughs in Language Model Training

Sara Kangaslahti, Elan Rosenfeld, Naomi Saphra

Keywords: interpretability techniques, loss disaggregation, phase transitions

TL;DR: We decompose changes in loss along an arbitrary basis of the low rank training subspace to find breakthroughs obscured in the aggregate loss

Abstract: The training loss curves of a neural network are typically smooth. Any visible discontinuities draw attention as discrete conceptual breakthroughs, while the rest of training is less carefully studied. In this work we hypothesize that similar breakthroughs actually occur frequently throughout training, though their presence is obscured when monitoring the aggregate train loss. To find these hidden transitions, we introduce POLCA, a method for decomposing changes in loss along an arbitrary basis of the low rank training subspace. We use our method to identify clusters of samples that exhibit similar changes in loss through training, disaggregating the overall loss into that of smaller groups of conceptually similar datapoints. We validate our method on synthetic arithmetic, showing that POLCA recovers clusters which represent easily interpretable breakthroughs in the model's capabilities whose existence would otherwise be lost in the crowd.

Submission Number: 100