Efficient Low-Dimensional Compression of Overparameterized Networks

Published: 20 Nov 2023, Last Modified: 01 Dec 2023CPAL 2024 (Recent Spotlight Track) PosterEveryoneRevisionsBibTeX
Keywords: overparameterization, deep networks, low-dimensional modeling
TL;DR: We present a simple, yet effective technique to compress overparameterized models and demonstrate its efficiency on solving several low-rank matrix recovery problems.
Abstract: Overparameterized models have proven to be powerful tools for solving various machine learning tasks. Their effectiveness is often attributed, at least in part, to the implicit bias inherent in their learning dynamics, which favors certain solutions that generalize well. This bias has particularly beneficial properties when learning low-rank models, such as reducing sample complexity and accelerating convergence. However, overparameterization often leads to a substantial increase in computational and memory costs, limiting the applicability of these models to real-world problems at scale. In this work, we aim to reduce this complexity by studying the compression of deep linear models. By extensively studying the learning dynamics of these models, we propose a simple, yet effective technique to compress deep linear networks that involve decreasing the width of the intermediate layers. Remarkably, we observe that with a particular choice of initialization, the compressed network converges faster than the original network, consistently yielding smaller recovery errors throughout all iterations of gradient descent. We substantiate this observation by developing a theory focused on deep matrix factorization problem, and by conducting empirical evaluations on two canonical matrix recovery problems: matrix sensing and completion. Further, we demonstrate how the use of compressed network can improve the efficiency of deep nonlinear networks. Overall, we observe that our compression technique accelerates the training process by more than $2\times$, without compromising model quality.
Track Confirmation: Yes, I am submitting to the recent spotlight track.
Submission Number: 54