Lipschitz Continuity in Deep Learning: A Systematic Review of Theoretical Foundations, Estimation Methods, Regularization Approaches and Certifiable Robustness

TMLR Paper5829 Authors

Abstract: Lipschitz continuity is a fundamental property of neural networks that characterizes their sensitivity to input perturbations. It plays a pivotal role in deep learning, governing robustness, generalization and optimization dynamics. Despite its importance, research on Lipschitz continuity is scattered across various domains, lacking a unified perspective. This paper addresses this gap by providing a systematic review of Lipschitz continuity in deep learning. We explore its theoretical foundations, estimation methods, regularization approaches, and certifiable robustness. By reviewing existing research through the lens of Lipschitz continuity, this survey serves as a comprehensive reference for researchers and practitioners seeking a deeper understanding of Lipschitz continuity and its implications in deep learning. Code: https://anonymous.4open.science/r/lipschitz_survey-DECE/

Submission Length: Long submission (more than 12 pages of main content)

Changes Since Last Submission:

We have revised the manuscript according to Reviewer X65t's comments as follows:

1. **Presentation Style.** Reviewer X65t raised a concern regarding the theorem/lemma like presentation. We have revised the presentation style. Now the distilled knowledge is presented in a consistent, story-driven manner. This directly addresses the reviewer’s concerns about presentation style. We also added new content for faciliating this presentation style. **Indeed, this revision required substantial effort over several weeks, but we welcomed the suggestion and fully embraced it as an opportunity to strengthen the quality of the manuscript.**

2. **Additional Literature.** We incorporated the references suggested by Reviewer X65t and integrated them into the relevant sections of the survey. Also, to reflect this suggestion, we realised we need to add a subsection under **Theoretical Foundations** for formally introducing **matrix orthogonalisation** for 1-Lipschitz.

3. **Refined References.** For each citation, we added precise locator information (e.g., theorem, lemma, section, or chapter) to facilitate easier navigation back to the original sources for readers. We also manually performed a consistency and accuracy check for all references.

4. **Other Revisions.** We corrected eg typos and addressed all other comments and suggestions raised by the reviewer(s).

Assigned Action Editor: ~Aurélien_Bellet1

Submission Number: 5829