A Unified Discretization Framework for Differential Equation Approach with Lyapunov Arguments for Convex Optimization

Kansei Ushiyama, Shun Sato, Takayasu Matsuo

Keywords: Convex optimization, Numerical analysis, Ordinary differential equations, Convergence estimate

TL;DR: We give a unified discretization method that has been desired in the differential equation approach with rate-revealing Lyapunov functionals for convex optimization.

Abstract:

The differential equation (DE) approach for convex optimization, which relates optimization methods to specific continuous DEs with rate-revealing Lyapunov functionals, has gained increasing interest since the seminal paper by Su--Boyd--Candès (2014).
However, the approach still lacks a crucial component to make it truly useful: there is no general, consistent way to transition back to discrete optimization methods. Consequently, even if we derive insights from continuous DEs, we still need to perform individualized and tedious calculations for the analysis of each method.
This paper aims to bridge this gap by introducing a new concept called ``weak discrete gradient'' (wDG), which consolidates the conditions required for discrete versions of gradients in the DE approach arguments.
We then define abstract optimization methods using wDG and provide abstract convergence theories that parallel those in continuous DEs.
We demonstrate that many typical optimization methods and their convergence rates can be derived as special cases of this abstract theory.
The proposed unified discretization framework for the differential equation approach to convex optimization provides an easy environment for developing new optimization methods and achieving competitive convergence rates with state-of-the-art methods, such as Nesterov's accelerated gradient.

Submission Number: 5877