Controlling the Complexity and Lipschitz Constant improves Polynomial Nets

Zhenyu Zhu, Fabian Latorre, Grigorios Chrysos, Volkan Cevher

Keywords: Polynomial Nets, Rademacher Complexity, Lipschitz constant, Coupled CP decomposition

Abstract: While the class of Polynomial Nets demonstrates comparable performance to neural networks (NN), it currently has neither theoretical generalization characterization nor robustness guarantees. To this end, we derive new complexity bounds for the set of Coupled CP-Decomposition (CCP) and Nested Coupled CP-decomposition (NCP) models of Polynomial Nets in terms of the $\ell_\infty$-operator-norm and the $\ell_2$-operator norm. In addition, we derive bounds on the Lipschitz constant for both models to establish a theoretical certificate for their robustness. The theoretical results enable us to propose a principled regularization scheme that we also evaluate experimentally and show that it improves the accuracy as well as the robustness of the models to adversarial perturbations. We showcase how this regularization can be combined with adversarial training, resulting in further improvements.

One-sentence Summary: We provide sample complexity results and bounds on the Lipschitz constant of polynomial networks, which we use to construct a regularization scheme that improves the robustness against adversarial noise.