Parallelized ADMM with General Objectives for Deep Learning

Yanqi Shi, Yu Tang, Hao Zheng, Zhigang Kan, Linbo Qiao

Abstract: While considerable efforts have been dedicated to improving models that employ regularized functions, the direct solution of non-convex models using most stochastic gradient optimization algorithms poses significant challenges due to their inherent non-convex nature. The Alternating Direction Method of Multipliers (ADMM) has emerged as a promising approach for addressing both convex and non-convex problems, boasting rapid convergence and effective constraint-handling capabilities. However, ADMM has not yet achieved significant advancements in the realm of non-convex regularized deep learning, and the development of parallelized ADMM techniques for non-convex objectives remains lacking. To address these challenges, this paper proposes the implementation of ADMM as a solution for solving general (non-convex regularized) deep learning tasks and presents a comprehensive analysis of its convergence properties. Furthermore, a parallelized framework for ADMM is proposed to address the absence of such advancements for general objectives. Experimental results reveal the stable convergence properties of ADMM when applied to non-convex objectives, demonstrating superior performance compared to ADMM with convex objectives. Additionally, we evaluate the computational efficiency of our proposed parallelized framework for ADMM.