Do Deeper Convolutional Networks Perform Better?

Eshaan Nichani, Adityanarayanan Radhakrishnan, Caroline Uhler

Keywords: Depth, Over-parameterization, Neural Networks

Abstract: Over-parameterization is a recent topic of much interest in the machine learning community. While over-parameterized neural networks are capable of perfectly fitting (interpolating) training data, these networks often perform well on test data, thereby contradicting classical learning theory. Recent work provided an explanation for this phenomenon by introducing the double descent curve, showing that increasing model capacity past the interpolation threshold leads to a decrease in test error. In line with this, it was recently shown empirically and theoretically that increasing neural network capacity through width leads to double descent. In this work, we analyze the effect of increasing depth on test performance. In contrast to what is observed for increasing width, we demonstrate through a variety of classification experiments on CIFAR10 and ImageNet-32 using ResNets and fully-convolutional networks that test performance worsens beyond a critical depth. We posit an explanation for this phenomenon by drawing intuition from the principle of minimum norm solutions in linear networks.

One-sentence Summary: In contrast to what is observed for increasing width, we demonstrate that test performance of convolutional neural networks worsens beyond a critical depth.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2010.09610/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=u6JCN9ny8