An Inexact Regularized Adaptive Algorithm with Manifold Identification for Training Structured Neural Networks

Zih-Syuan Huang, Ching-pei Lee

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Deep learning, structured models, adaptive method, manifold identification, variance reduction, inexact subproblem solution

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: An inexact regularized adaptive dual averaging algorithm with momentum for training structured neural networks, with outstanding performance in computer vision, language, and speech tasks.

Abstract: We propose an inexact regularized adaptive dual averaging algorithm with momentum, RAMDA, for training structured neural networks in various tasks with the help of regularization. Through the theory of manifold identification, we show that after a finite number of steps, the structures of the iterates generated by RAMDA are all identical to the structure induced by the regularization at the stationary point of asymptotic convergence. This structure is locally optimal within a neighborhood of the point of convergence and therefore provides the best possible performance among all methods converging to the same point. To make use of manifold identification, RAMDA produces stochastic estimators of the gradient that almost surely converge to the true gradient even when the training problem is no longer a finite-sum one but a stochastic one over a certain probability distribution due to data augmentation. With the simultaneous presence of a preconditioner and a regularization term, the subproblem of RAMDA as well as those of existing frameworks have no closed-form solutions, so we also propose a general iterative subroutine for approximately solving such subproblems efficiently while maintaining similar convergence guarantees. Extensive numerical experiments in modern computer vision, natural language processing, and speech tasks show that our subproblem solver is efficient and applicable to existing frameworks, and the proposed RAMDA excels state of the art for training structured neural networks to generate more structural points without decreasing the prediction performance.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3314