Robust low-rank training via approximate orthonormal constraints

Dayana Savostianova, Emanuele Zangrando, Gianluca Ceruti, Francesco Tudisco

Keywords: low-rank neural networks, Stiefel manifold, orthogonal neural networks, pruning, adversarial robustness, neural network condition number, neural network singular values

TL;DR: We observe that low-rank layers may negatively affect network stability and we propose an algorithm that mitigates this phenomenon in a computationally affordable way, while maintaining the low-rank structure.

Abstract:

With the growth of model and data sizes, a broad effort has been made to design pruning techniques that reduce the resource demand of deep learning pipelines, while retaining model performance. In order to reduce both inference and training costs, a prominent line of work uses low-rank matrix factorizations to represent the network weights. Although able to retain accuracy, we observe that low-rank methods tend to compromise model robustness against adversarial perturbations. By modeling robustness in terms of the condition number of the neural network, we argue that this loss of robustness is due to the exploding singular values of the low-rank weight matrices. Thus, we introduce a robust low-rank training algorithm that maintains the network's weights on the low-rank matrix manifold while simultaneously enforcing approximate orthonormal constraints. The resulting model
reduces both training and inference costs while ensuring well-conditioning and thus better adversarial robustness, without compromising model accuracy. This is shown by extensive numerical evidence and by our main approximation theorem that shows the computed robust low-rank network well-approximates the ideal full model, provided a highly performing low-rank sub-network exists.

Submission Number: 7788