On quantizing the state of the Muon optimizer

ICLR 2026 Conference Submission22234 Authors

Keywords: optimization, muon, llms, quantization

TL;DR: An 8-bit quantized version of the Muon optimizer saves memory and is competitive in training

Abstract: The Muon optimizer, based on matrix orthogonalization, has recently shown faster convergence and up to 2× computational efficiency over AdamW in LLM pretraining. Like AdamW, Muon is stateful, requiring storage of both model weights and accumulated gradients. While 8-bit AdamW variants mitigate this overhead using blockwise quantization, they are typically stable only under dynamic quantization - which improves stability on linear quantization for extreme values. In this paper, we introduce the 8-bit Muon optimizer using blockwise quantization, supporting both linear and dynamic schemes. We demonstrate that 8-bit Muon maintains stability under both, while delivering $\sim$74\% reduction in memory footprint compared to full-precision Muon. In extensive experiments, 8-bit Muon closely matches the performance of Muon while outperforming AdamW and 8-bit AdamW in pre-training a 1.6B model on 4B FineWeb tokens. It also shows competitive results when fine-tuning the Llama 3.2 3B model on post-training data. We also provide a theoretical perspective to help explain this robustness under quantization.

Primary Area: optimization

Submission Number: 22234