Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Supervised Learning

Florian Bordes, Randall Balestriero, Quentin Garrido, Adrien Bardes, Pascal Vincent

Abstract:

One unexpected technique that emerged in recent years consists in training a Deep Network (DN) with a Self-Supervised Learning (SSL) method, and using this network on downstream tasks but with its last few layers entirely removed. This usually skimmed-over trick of throwing away the entire projector is actually critical for SSL methods to display competitive performances. For example, on ImageNet classification, more than 30 points of percentage can be gained that way. This is a little vexing, as one would hope that the network layer at which invariance is explicitly enforced by the SSL criterion during training (the last layer) should be the one to use for best generalization performance downstream. But it seems not to be, and this study sheds some light on why.
This trick, which we name Guillotine Regularization (GR), is in fact a generically applicable method that has been used to improve generalization performance in transfer learning scenarios. In this work, we identify the underlying reasons behind its success and challenge the preconceived idea that we should through away the entire projector in SSL. In fact, the optimal layer to use might change significantly depending on the training setup, the data or the downstream task. Lastly, we give some insights on how to reduce the need for a projector in SSL by aligning the pretext SSL task and the downstream task.

Submission Length: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Jinwoo_Shin1

License: Creative Commons Attribution 4.0 International (CC BY 4.0)

Submission Number: 757