Layerwise complexity-matched learning yields an improved model of cortical area V2
Abstract: Human ability to recognize complex visual patterns arises through transformations performed by successive areas in the ventral visual cortex. Deep neural networks trained end-to-end for object recognition approach human capabilities, and offer the best descriptions to date of neural responses in the late stages of the hierarchy. But these networks provide a poor account of the early stages, compared to traditional hand-engineered models, or models optimized for coding efficiency or prediction. Moreover, the gradient backpropagation used in end-to-end learning is generally considered to be biologically implausible. Here, we overcome both of these limitations by developing a bottom-up self-supervised training methodology that operates independently on successive layers. Specifically, we maximize feature similarity between pairs of locally-deformed natural image patches, while decorrelating features across patches sampled from other images. Crucially, the deformation amplitudes are adjusted proportionally to receptive field sizes in each layer, thus matching the task complexity to the capacity at each stage of processing. In comparison with architecture-matched versions of previous models, we demonstrate that our layerwise complexity-matched learning (LCL) formulation produces a two-stage model (LCL-V2) that is better aligned with selectivity properties and neural activity in primate area V2. We demonstrate that the complexity-matched learning paradigm is critical for the emergence of the improved biological alignment. Finally, when the two-stage model is used as a fixed front-end for a deep network trained to perform object recognition, the resultant model (LCL-V2Net) is significantly better than standard end-to-end self-supervised, supervised, and adversarially-trained models in terms of generalization to out-of-distribution tasks and alignment with human behavior.
Submission Length: Long submission (more than 12 pages of main content)
Changes Since Last Submission: Addressed comments from all reviewers:
- added statistical significance results to texture modulation results and fixed bug in texture modulation plots
- added error bars to BrainScore prediction plots
- added per-neuron BrainScore explained variance plots to compare our model against others
- miscellaneous edits to the text to clean up explanations of the methodology and typos
revision 2: added new statistical significance test, specifically testing significance of difference in rank correlations between models. Also re-calculated texture modulation statistics removing noisy biological neurons.
Assigned Action Editor: ~Robert_Legenstein1
Submission Number: 1909
Loading