An Axiomatic Atlas for Optimization

Abstract: First-order methods are the workhorses of modern large-scale optimization, powering training and inference across machine learning, signal processing, and scientific computing. Yet the theoretical guarantees that explain their behavior are dispersed across smooth, nonsmooth, stochastic, and composite settings, while practitioners must choose among many algorithmic variants and tune interacting hyperparameters with limited guidance about which assumptions actually matter. We introduce the Optimization Atlas, an axiomatic view that organizes widely used first-order methods and their canonical convergence behaviors within a single, explicit assumption space. The atlas exposes inclusion-minimal assumption sets that suffice for a desired outcome, and it delineates sharp frontiers where a single assumption change alters the attainable regime, such as sublinear versus linear convergence or linear convergence versus variance-limited floors. We then leverage the induced theorem by axiom structure to uncover a small number of recurring modes that clarify which phenomena are structurally shared across methods and which correspond to genuinely distinct mechanisms. Finally, we convert the atlas into a practical diagnostic control plane: from short training traces it estimates the active limiting ceiling and ranks interventions, ranging from relaxing modeling assumptions (for example via smoothing or regularization) to increasing algorithmic capacity (for example via batching or variance reduction). Experiments on controlled synthetic problems and a CIFAR-10 convolutional network show that this control plane reliably identifies the governing regime, recommends high-return changes when optimization is the bottleneck, and abstains when additional tuning is unlikely to help.

Submission Type: Long submission (more than 12 pages of main content)

Assigned Action Editor: ~Guillaume_Dalle1

Submission Number: 7776