Locally connected networks as ventral stream models

Roman Pogodin, Peter E. Latham

Keywords: locally connected networks, convolutional network

Abstract: Most deep learning models of the ventral stream, and convolutional networks in particular, share weights among neurons. Weight sharing during learning is crucial for good performance on image recognition tasks, but it is not biologically plausible. In this work, we compare performance and Brain-Score results of ImageNet-trained networks in multiple configurations: convolutional, locally connected (i.e., convolutional without weight sharing), and locally connected with anti-Hebbian plasticity mechanisms that promote weight sharing. We also study the role of initialization on performance of those networks. We find that the more weight sharing networks have, the better they perform on both ImageNet and Brain-Score, which can sometimes be further improved with a convolutional initialization. However, locally connected networks outperform their convolutional counterparts on purely neural data (areas V1, V2, V4, IT), but not on behavioral responses. Moreover, ImageNet performance negatively correlates with correspondence to V1 data, suggesting that better models of early visual processing don't necessarily provide a good input for models of deeper visual areas.

TL;DR: locally connected networks are better fit V1 than conv nets, depite worse overall performance