Modeling Visual Cortex by Maximizing Layerwise Multiscale Manifold Capacity

Thomas Edward Yerxa, SueYeon Chung, Eero P Simoncelli

Keywords: Biological Plausibility, Efficient Coding, Visual Cortex, Layerwise Learning, Self-supervised Learning

Abstract: Task-optimized deep neural networks have risen to prominence as the most predictive phenomenological models of responses in primate visual cortex, but leave much to be desired from the perspective of biological plausibility. One such limitation is the reliance on precise credit assignment through global backpropagation of error signals. Recent work has shown that this weakness can be circumvented by requiring each subsequent stage to solve a distinct and increasingly complex task, allowing for layerwise local learning signals. We propose a novel strategy for crafting such intermediate losses that uses an efficient coding framework formulated in terms of manifold capacity, which can be computed using a sequence of *canonical cortical computations*. In particular, we leverage the relationship between the multiscale nature of visual signals and the dilation of receptive field sizes in cascaded visual representations to modulate complexity, allowing for the reapplication of these common loss computations at each stage of the hierarchy. We evaluate our approach on its ability to predict neural datasets spanning three areas of the ventral stream hierarchy in macaque, as well as human psychophysical data on an object classification task. We find that our unsupervised layerwise model matches or exceeds the performance of competitive architecture-matched baselines on all evaluations considered.

Primary Area: applications to neuroscience & cognitive science

Submission Number: 14119