Test like you Train in Implicit Deep Learning

Zaccharie Ramzi, Pierre Ablin, Gabriel Peyré, Thomas Moreau

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: deep equilibrium models, implicit differentiation, bilevel optimization, bilevel, implicit deep learning, meta-learning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We show that Deep Equilibrium Models (DEQs) do not in practice benefit from a higher number of inner iterations at test-time compared to that used in training.

Abstract: Implicit deep learning has recently gained popularity with applications ranging from meta-learning to Deep Equilibrium Networks~(DEQs). In its very general formulation, it relies on expressing some components of deep learning pipelines implicitly, typically via a root equation called the inner problem. In practice, the solution of the inner problem is approximated with an iterative procedure, usually with a fixed number of inner iterations during training. At inference time, the inner problems needs to be solved with new data. A popular belief is that increasing the number of inner iterations relative to the one used in training yields better performances. In this paper, we question such an assumption and provide a detailed theoretical analysis in a simple affine setting. We demonstrate that overparametrization plays a key role: increasing the number of iterations at test time cannot improve performances for overparametrized networks. We validate our theory on an array of implicit deep-learning problems. We show that DEQs, which are typically overparametrized, do not benefit from increasing the number of iterations at inference while meta-learning, which is typically not overparametrized, benefits from it.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 150