On the inductive bias of infinite-depth ResNets and the bottleneck rank

Enric Boix-Adserà

Abstract: We compute the minimum-norm weights of a deep linear ResNet, and find that the inductive bias of this architecture lies between minimizing nuclear norm and rank. This implies that, with appropriate hyperparameters, deep nonlinear ResNets have an inductive bias towards minimizing bottleneck rank.