Understanding CNNs as a model of the inferior temporal cortex: using mediation analysis to unpack the contribution of perceptual and semantic features in random and trained networks

Anna Truzzi, Rhodri Cusack

Keywords: CNNs, semantic, perceptual, IT, inferior temporal cortex, visual brain model, random network, trained network

TL;DR: Random and trained CNNs can predict IT activity with similar magnitude but for different reasons: the former captures its perceptual features, the latter captures its semantic aspects only, having discarded perceptual information relevant to IT.

Abstract:

Convolutional neural networks (CNNs) trained for visual recognition can predict activity in primate inferior temporal cortex (IT). It was generally accepted that this is because training leads the CNNs to develop tuning to visual features similar to those in the brain. However, recent evidence that untrained random-weight CNNs explain IT variance to a similar magnitude appears inconsistent with this view.
Since IT contains rich representations of both perceptual and semantic features, here we propose a resolution to this conflict, that random and trained networks capture different aspects of IT activity. Specifically, we hypothesised that random networks capture perceptual aspects of IT, while trained networks capture semantic aspects but not perceptual ones.
We evaluated a trained standard AlexNet and an untrained random network shown to correlate better with the brain, DeepCluster. The ability of the CNNs to predict IT activity patterns and the role played by perceptual and semantic features was evaluated using regression models, multidimensional scaling, and mediation analysis.
The results support the hypothesis and highlight that, whether CNNs are used as models of the brain, or the brain is used to inspire advances in neural networks, it is not enough to know how similar a given model is to the brain: we also need to know why.