Gradient Flow Convergence Guarantee for General Neural Network Architectures

Yash Jakhmola

Keywords: gradientflow, linearconvergence, optimization

TL;DR: This paper provides a unified proof of linear convergence of gradient flow for optimizing neural networks with piecewise non-zero polynomial activation or ReLU, sigmoid activation functions.

Abstract: A key challenge in modern deep learning theory is to explain the remarkable success of gradient-based optimization methods when training large-scale, complex deep neural networks. Though linear convergence of such methods has been proved for a handful of specific architectures, a united theory still evades researchers. This article presents a unified proof for linear convergence of continuous gradient descent, also called gradient flow, while training any neural network with piecewise non-zero polynomial activations or ReLU, sigmoid activations. Our primary contribution is a single, general theorem that not only covers architectures for which this result was previously unknown but also consolidates existing results under weaker assumptions. While our focus is theoretical and our results are only exact in the infinitesimal step size limit, we nevertheless find excellent empirical agreement between the predictions of our result and those of the practical step-size gradient descent method.

Primary Area: optimization

Submission Number: 11537