Meta-Optimizing ML Model Training

ICLR 2026 Conference Submission18696 Authors

Keywords: data optimizaton, data selection, data poisoning

Abstract: A major challenge in training large-scale machine learning models is configuring the training process to maximize model performance, i.e., finding the best training setup from a vast design space. In this work, we unlock a gradient-based approach to this problem. We first introduce an algorithm for efficiently calculating metagradients---gradients through model training---at scale. We then introduce a "smooth model training" framework that enables effective optimization using metagradients. With metagradient descent (MGD), we, e.g., greatly improve on existing dataset selection methods and outperform accuracy-degrading data poisoning attacks by an order of magnitude.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 18696