Adam through a Second-Order Lens

Ross M Clarke, Baiyu Su, José Miguel Hernández-Lobato

Keywords: Optimisation for Neural Networks, Second-Order Optimisation

TL;DR: We combine second-order heuristics from K-FAC with the effective update directions of Adam to produce a hybrid optimiser, from which interesting conclusions can be drawn.

Abstract: Research into optimisation for deep learning is characterised by a tension between the computational efficiency of first-order, gradient-based methods (such as SGD and Adam) and the theoretical efficiency of second-order, curvature-based methods (such as quasi-Newton methods and K-FAC). We seek to combine the benefits of both approaches into a single computationally-efficient algorithm. Noting that second-order methods often depend on stabilising heuristics (such as Levenberg-Marquardt damping), we propose AdamQLR: an optimiser combining damping and learning rate selection techniques from K-FAC with the update directions proposed by Adam, inspired by considering Adam through a second-order lens. We evaluate AdamQLR on a range of regression and classification tasks at various scales, achieving competitive generalisation performance vs runtime.

Submission Number: 60