Lipschitz Continuity in Deep Learning: A Systematic Review of Theoretical Foundations, Estimation Methods, Regularization Approaches and Certifiable Robustness

TMLR Paper5829 Authors

Abstract: Lipschitz continuity is a fundamental property of neural networks that characterizes their sensitivity to input perturbations. It plays a pivotal role in deep learning, governing robustness, generalization and optimization dynamics. Despite its importance, research on Lipschitz continuity is scattered across various domains, lacking a unified perspective. This paper addresses this gap by providing a systematic review of Lipschitz continuity in deep learning. We explore its theoretical foundations, estimation methods, regularization approaches, and certifiable robustness. By reviewing existing research through the lens of Lipschitz continuity, this survey serves as a comprehensive reference for researchers and practitioners seeking a deeper understanding of Lipschitz continuity and its implications in deep learning. Code: https://anonymous.4open.science/r/lipschitz_survey-DECE/

Submission Length: Long submission (more than 12 pages of main content)

Changes Since Last Submission:

We have revised the manuscript according to the requests from Reviewer X65t and Reviewer Rbf2 as follows:

## Revisions regarding reviews from Reviewer X65t
1. **Presentation Style.** Reviewer X65t raised a concern regarding the theorem/lemma like presentation. We have revised the presentation style. Now the distilled knowledge is presented in a consistent, story-driven manner. This directly addresses the reviewer’s concerns about presentation style. We also added new content for faciliating this presentation style. **Indeed, this revision required substantial effort over several weeks, but we welcomed the suggestion and fully embraced it as an opportunity to strengthen the quality of the manuscript.**

2. **Additional Literature.** We incorporated the references suggested by Reviewer X65t and integrated them into the relevant sections of the survey. Also, to reflect this suggestion, we realised we need to add a subsection under **Theoretical Foundations** for formally introducing **matrix orthogonalisation** for 1-Lipschitz.

3. **Refined References.** For each citation, we added precise locator information (e.g., theorem, lemma, section, or chapter) to facilitate easier navigation back to the original sources for readers. We also manually performed a consistency and accuracy check for all references.

4. **Other Revisions.** We corrected eg typos and addressed all other comments and suggestions raised by the reviewer(s).

## Revisions regarding reviews from Reviewer Rbf2

1. **Notation Inconsistencies**. We have revised the notations to improve consistency throughout the manuscript.

2. **Typos and Errors**. We have corrected the typos and errors pointed out by Reviewer Rbf2.

3. **Content Restructuring**. We have reorganized the material related to the Lipschitz continuity of modern transformer architectures for improved clarity and coherence.

4. **Suppressing Equation Numbering.** We removed some unnecessary equation numberings as suggested in particular in proofs.

5. **Enhanced Citations**. We have further strengthened the citations by explicitly referencing relevant definitions, lemmas, and theorems to better guide the reader.

6. **Sanity Check for References**. To ensure the survey quality, we also conducted an additional round of sanity checks on the references to ensure the appropriateness and quality of the associated content.

7. **Generalization bound**. We have revised the generalisation bound by using vector contraction lemma (Maurer, 2016, Equation 1 & Corollary 1).

Assigned Action Editor: ~Aurélien_Bellet1

Submission Number: 5829