Ano : Faster is Better in Noisy Lanscapes

Adrien Kegreisz

Keywords: Stochastic optimization, Adaptive optimizers, Reinforcement learning, First-order methods

TL;DR: We present Ano, a stochastic optimizer that separates direction and magnitude to improve stability in noisy and non-stationary settings, supported by theoretical analysis and empirical evaluation.

Abstract: Stochastic optimizers are central to deep learning, yet widely used methods such as Adam and Adan can degrade in non-stationary or noisy environments, partly due to their reliance on momentum-based magnitude estimates. We introduce Ano, a novel optimizer that decouples direction and magnitude: momentum is used for directional smoothing, while instantaneous gradient magnitudes determine step size. This design improves robustness to gradient noise while retaining the simplicity and efficiency of first-order methods. We further propose Anolog, which removes sensitivity to the momentum coefficient by expanding its window over time via a logarithmic schedule. We establish non-convex convergence guarantees with a convergence rate similar to other sign-based methods, and empirically show that Ano provides substantial gains in noisy and non-stationary regimes such as reinforcement learning, while remaining competitive on low-noise tasks.

Primary Area: optimization

Submission Number: 4121