Assessing the scalability of biologically-motivated deep learning algorithms and architectures

Sergey Bartunov, Adam Santoro, Blake A. Richards, Geoffrey E. Hinton, Timothy Lillicrap

Abstract: The backpropagation of error algorithm (BP) is often said to be impossible to implement in a real brain. The recent success of deep networks in machine learning and AI, however, has inspired a number of proposals for understanding how the brain might learn across multiple layers, and hence how it might implement or approximate BP. As of yet, none of these proposals have been rigorously evaluated on tasks where BP-guided deep learning has proved critical, or in architectures more structured than simple fully-connected networks. Here we present the first results on scaling up a biologically motivated model of deep learning to datasets which need deep networks with appropriate architectures to achieve good performance. We present results on CIFAR-10 and ImageNet. For CIFAR-10 we show that our algorithm, a straightforward, weight-transport-free variant of difference target-propagation (DTP) modified to remove backpropagation from the penultimate layer, is competitive with BP in training deep networks with locally defined receptive fields that have untied weights. For ImageNet we find that both DTP and our algorithm perform significantly worse than BP, opening questions about whether different architectures or algorithms are required to scale these approaches. Our results and implementation details help establish baselines for biologically motivated deep learning schemes going forward.

TL;DR: Benchmarks for biologically plausible learning algorithms on complex datasets and architectures

Keywords: target propagation, biologically-plausible learning, benchmark, neuroscience

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/assessing-the-scalability-of-biologically/code)