Biologically-Plausible Learning Algorithms Can Scale to Large Datasets

Will Xiao, Honglin Chen, Qianli Liao, Tomaso Poggio

Abstract: The backpropagation (BP) algorithm is often thought to be biologically implausible in the brain. One of the main reasons is that BP requires symmetric weight matrices in the feedforward and feedback pathways. To address this “weight transport problem” (Grossberg, 1987), two biologically-plausible algorithms, proposed by Liao et al. (2016) and Lillicrap et al. (2016), relax BP’s weight symmetry requirements and demonstrate comparable learning capabilities to that of BP on small datasets. However, a recent study by Bartunov et al. (2018) finds that although feedback alignment (FA) and some variants of target-propagation (TP) perform well on MNIST and CIFAR, they perform significantly worse than BP on ImageNet. Here, we additionally evaluate the sign-symmetry (SS) algorithm (Liao et al., 2016), which differs from both BP and FA in that the feedback and feedforward weights do not share magnitudes but share signs. We examined the performance of sign-symmetry and feedback alignment on ImageNet and MS COCO datasets using different network architectures (ResNet-18 and AlexNet for ImageNet; RetinaNet for MS COCO). Surprisingly, networks trained with sign-symmetry can attain classification performance approaching that of BP-trained networks. These results complement the study by Bartunov et al. (2018) and establish a new benchmark for future biologically-plausible learning algorithms on more difficult datasets and more complex architectures.

Keywords: biologically plausible learning algorithm, ImageNet, sign-symmetry, feedback alignment

TL;DR: Biologically plausible learning algorithms, particularly sign-symmetry, work well on ImageNet

Code: [ willwx/sign-symmetry](https://github.com/willwx/sign-symmetry) + [ 1 community implementation](https://paperswithcode.com/paper/?openreview=SygvZ209F7)

Data: [ImageNet](https://paperswithcode.com/dataset/imagenet)

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:1811.03567/code)