A Novel Unified Parametric Assumption for Nonconvex Optimization

Artem Riabinin, Ahmed Khaled, Peter Richtárik

Keywords: Nonconvex Optimization, Nonconvex Assumptions

TL;DR: We propose a flexible parametric assumption that captures many nonconvex problems and delivers unified convergence guarantees for gradient-based methods in both deterministic and stochastic settings.

Abstract: Nonconvex optimization is central to modern machine learning, but the general framework of nonconvex optimization yields weak convergence guarantees that are too pessimistic compared to practice. On the other hand, while convexity enables efficient optimization, it is of limited applicability to many practical problems. To bridge this gap and better understand the practical success of optimization algorithms in nonconvex settings, we introduce a novel unified parametric assumption. Our assumption is general enough to encompass a broad class of nonconvex functions while also being specific enough to enable the derivation of a unified convergence theorem for gradient-based methods. Notably, by tuning the parameters of our assumption, we demonstrate its versatility in recovering several existing function classes as special cases and in identifying functions amenable to efficient optimization. We derive our convergence theorem for both deterministic and stochastic optimization, and conduct experiments to verify that our assumption can hold practically over optimization trajectories.

Primary Area: optimization

Submission Number: 11986