No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models

Jean Kaddour, Oscar Key, Piotr Nawrot, Pasquale Minervini, Matt Kusner

Keywords: language models, transformers, efficient training

TL;DR: We investigate efficient training algorithms for Transformer Language models and find many situations in which they have only marginal improvements over standard training.

Abstract:

The computation necessary for training Transformer-based language models has skyrocketed in recent years.
This trend has motivated research on efficient training algorithms designed to improve training, validation, and downstream performance faster than standard training. In this work, we revisit three categories of such algorithms: dynamic architectures (layer stacking, layer dropping), batch selection (selective backprop., RHO-loss), and efficient optimizers (Lion, Sophia). When pre-training BERT and T5 with a fixed computation budget using such methods, we find that their training, validation, and downstream gains vanish compared to a baseline with a fully-decayed learning rate. We define an evaluation protocol that enables computation to be done on arbitrary machines by mapping all computation time to a reference machine which we call reference system time. We discuss the limitations of our proposed protocol and release our code to encourage rigorous research in efficient training procedures: https://github.com/JeanKaddour/NoTrainNoGain.

Submission Number: 15048