Exploiting Connections between Lipschitz Structures for Certifiably Robust Deep Equilibrium Models

Aaron J Havens, Alexandre Araujo, Siddharth Garg, Farshad Khorrami, Bin Hu

Keywords: Deep equilibrium models, Lipschitz networks, certified robustness

TL;DR: We delineate the connections between various Lipschitz parameterizations of DEQ and existing state-of-the-art explicit Lipschitz networks to improve our understanding of achieving better certified robustness for DEQ.

Abstract: Recently, deep equilibrium models (DEQs) have drawn increasing attention from the machine learning community. However, DEQs are much less understood in terms of certified robustness than their explicit network counterparts. In this paper, we advance the understanding of certified robustness of DEQs via exploiting the connections between various Lipschitz network parameterizations for both explicit and implicit models. Importantly, we show that various popular Lipschitz network structures, including convex potential layers (CPL), SDP-based Lipschitz layers (SLL), almost orthogonal layers (AOL), Sandwich layers, and monotone DEQs (MonDEQ) can all be reparameterized as special cases of the Lipschitz-bounded equilibrium networks (LBEN) without changing the prescribed Lipschitz constant in the original network parameterization. A key feature of our reparameterization technique is that it preserves the Lipschitz prescription used in different structures. This opens the possibility of achieving improved certified robustness of DEQs via a combination of network reparameterization, structure-preserving regularization, and LBEN-based fine-tuning. We also support our theoretical understanding with new empirical results, which show that our proposed method improves the certified robust accuracy of DEQs on classification tasks. All codes and experiments are made available at \url{https://github.com/AaronHavens/ExploitingLipschitzDEQ}.

Submission Number: 9248