Combining Explicit and Implicit Regularization for Efficient Learning in Deep NetworksDownload PDF

Published: 31 Oct 2022, Last Modified: 05 Jan 2023NeurIPS 2022 AcceptReaders: Everyone
Keywords: optimization, regularization, deep learning, matrix factorization, neural networks, efficient learning, implicit regularization, explicit regularization, Matrix Completion, Optimization for Deep Networks
TL;DR: Explicit regluarization and optimizer's own inductive biases can combine to form beneficial effects for efficient learning
Abstract: Works on implicit regularization have studied gradient trajectories during the optimization process to explain why deep networks favor certain kinds of solutions over others. In deep linear networks, it has been shown that gradient descent implicitly regularizes toward low-rank solutions on matrix completion/factorization tasks. Adding depth not only improves performance on these tasks but also acts as an accelerative pre-conditioning that further enhances this bias towards low-rankedness. Inspired by this, we propose an explicit penalty to mirror this implicit bias which only takes effect with certain adaptive gradient optimizers (e.g. Adam). This combination can enable a degenerate single-layer network to achieve low-rank approximations with generalization error comparable to deep linear networks, making depth no longer necessary for learning. The single-layer network also performs competitively or out-performs various approaches for matrix completion over a range of parameter and data regimes despite its simplicity. Together with an optimizer’s inductive bias, our findings suggest that explicit regularization can play a role in designing different, desirable forms of regularization and that a more nuanced understanding of this interplay may be necessary.
Supplementary Material: zip
18 Replies