Computational models of vision have traditionally been developed in a bottom-up fashion, by hierarchically composing a series of straightforward operations - i.e. convolution and pooling  - with the aim of emulating simple and complex cells in the visual cortex, resulting in the introduction of deep convolutional neural networks (CNNs). Nevertheless, data obtained with recent neuronal recording techniques support that the nature of the computations carried out in the ventral visual stream is not completely captured by current deep CNN models. To fill the gap between the ventral visual stream and deep models, several benchmarks have been designed and organized into the Brain-Score platform, granting a way to perform multi-layer (V1, V2, V4, IT) and behavioral comparisons between the two counterparts. In our work, we aim to shift the focus on architectures that take into account lateral recurrent connections, a ubiquitous feature of the ventral visual stream, to devise adaptive receptive fields. Through recurrent connections, the input’s long-range spatial dependencies can be captured in a local multi-step fashion and, as introduced with Gated Recurrent CNNs (GRCNN), the unbounded expansion of the neuron’s receptive fields can be modulated through the use of gates. In order to increase the robustness of our approach and the biological fidelity of the activations, we employ specific data augmentation techniques in line with several of the scoring benchmarks. Enforcing some form of invariance, through heuristics, was found to be beneficial for better neural predictivity.

Improving Neural Predictivity in the Visual Cortex with Gated Recurrent Connections

Most deep learning models of the ventral stream, and convolutional networks in particular, share weights among neurons. Weight sharing during learning is crucial for good performance on image recognition tasks, but it is not biologically plausible. In this work, we compare performance and Brain-Score results of ImageNet-trained networks in multiple configurations: convolutional, locally connected (i.e., convolutional without weight sharing), and locally connected with anti-Hebbian plasticity mechanisms that promote weight sharing. We also study the role of initialization on performance of those networks. We find that the more weight sharing networks have, the better they perform on both ImageNet and Brain-Score, which can sometimes be further improved with a convolutional initialization. However, locally connected networks outperform their convolutional counterparts on purely neural data (areas V1, V2, V4, IT), but not on behavioral responses. Moreover, ImageNet performance negatively correlates with correspondence to V1 data, suggesting that better models of early visual processing don't necessarily provide a good input for models of deeper visual areas.

Locally connected networks as ventral stream models

The human-level performance of artificial neural networks (ANNs) in visual processing has made them a much-used research subject for understanding how the visual cortex really works. To assess how well various types of ANNs represent regions of the visual cortex, the Brain-Score platform provides several standardized benchmarks. These include the measure of explained variance in ventral stream regions V1, V2, V4, IT and the object recognition behavior in primates.
The aim of this work is to find a training procedure that maximizes an ANNs average score in the Brain-Score benchmark. The proposed pipeline combines a customized version of CutMix, heavy use of image augmentations, adversarial robust training, fixing the train-test resolution discrepancy, and weight averaging. 
Due to its widespread use, memory and computational efficiency, and object recognition performance, the EfficientNet-B1 architecture was used prototypically. The proposed training methods improve the public object recognition behavior metric score by 9% and the explained V1 variance by 62%, resulting in the best performing models in the Brain-Score competition 2022 . This is a strong indicator that finding the right training strategy might be crucial for developing brain-like ANNs.

Bag of Tricks for Training Brain-Like Deep Neural Networks

Modern high-scoring models of vision in the brain score competition do not stem from Vision Transformers. However, in this short paper, we provide evidence against the unexpected trend of Vision Transformers (ViT) being not perceptually aligned with human visual representations by showing how a dual-stream Transformer, a CrossViT$~\textit{a la}$ Chen et al. (2021), under a joint rotationally-invariant and adversarial optimization procedure yields 2nd place in the aggregate Brain-Score 2022 competition averaged across all visual categories, and currently holds the 1st place for the highest explainable variance of area V4. In addition, our current Transformer-based model also achieves greater explainable variance for areas V4, IT, and Behaviour than a biologically-inspired CNN (ResNet50) that integrates a frontal V1-like computation module (Dapello et al., 2020). Our team was also the only entry in the top-5 that shows a positive rank correlation between explained variance per area and depth in the visual hierarchy. Against our initial expectations, these results provide tentative support for an $\textit{``All roads lead to Rome''}$ argument enforced via a joint optimization rule even for non biologically-motivated models of vision such as Vision Transformers. Code is available at https://github.com/williamberrios/BrainScore-Transformers

Brain-Score 2022 Workshop BSW Submissions

Improving Neural Predictivity in the Visual Cortex with Gated Recurrent Connections

Locally connected networks as ventral stream models

Bag of Tricks for Training Brain-Like Deep Neural Networks

Joint rotational invariance and adversarial training of a dual-stream Transformer yields state of the art Brain-Score for Area V4