Towards Second-Order Optimization in Learned Image Compression: Faster, Better, and More Deployable

ICLR 2026 Conference Submission626 Authors

Keywords: Learned Image Compression, Training Efficiency

Abstract: Training learned image compression (LIC) models entails navigating a challenging optimization landscape defined by the fundamental trade-off between rate and distortion. Standard first-order optimizers, such as SGD and Adam, struggle with gradient conflicts arising from competing objectives, leading to slow convergence and suboptimal rate–distortion performance. In this work, we demonstrate that a simple switch to a second-order quasi-Newton optimizer, SOAP, dramatically improves both training efficiency and final performance across diverse LIC architectures. Our theoretical and empirical analyses reveal that SOAP’s Newton preconditioning inherently resolves the intra-step and inter-step update conflicts intrinsic to the R–D objective, facilitating faster, more stable convergence. Beyond acceleration, we uncover a critical deployability benefit: SOAP-trained (non-diagonal) models exhibit significantly fewer activation and latent outliers. This improves entropy modeling and substantially enhances robustness to post-training quantization. Together, these results establish second-order optimization—achievable as a seamless drop-in replacement of the imported optimizer—as a powerful, practical tool for advancing the efficiency and real-world readiness of LICs. Code will be publicly available.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 626