A provable control of sensitivity of neural networks through a direct parameterization of the overall bi-Lipschitzness

Yuri Kinoshita, Taro Toyoizumi

Keywords: bi-Lipschitzness, theoretical guarantee, tight control, direct parameterization, inductive bias, convex neural network, Legendre-Fenchel transformation

Abstract: While neural networks can enjoy an outstanding flexibility and exhibit unprecedented performance, the mechanism behind their behavior is still not well-understood. To tackle this fundamental challenge, researchers have tried to restrict and manipulate some of their properties in order to gain new insights and better control on them. Especially, throughout the past few years, the concept of *bi-Lipschitzness* has been proved as a beneficial inductive bias in many areas. However, due to its complexity, the design and control of bi-Lipschitz architectures are falling behind, and a model that is precisely designed for bi-Lipschitzness realizing a direct and simple control of the constants along with solid theoretical analysis is lacking. In this work, we investigate and propose a novel framework for bi-Lipschitzness that can achieve such a clear and tight control based on convex neural networks and the Legendre-Fenchel duality. Its desirable properties are illustrated with concrete experiments to illustrate its broad range of applications.

Primary Area: Learning theory

Submission Number: 6258