Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures

Sergey Bartunov, Adam Santoro, Blake A. Richards, Geoffrey E. Hinton, Timothy Lillicrap

Keywords: biologically-plausible learning methods, deep learning, neuroscience, target propagation

TL;DR: Do biologically-plausible learning methods such as target propagation or feedback alignment scale to ImageNet?

Abstract: The backpropagation of error algorithm (BP) is often said to be impossible to implement in a real brain. The recent success of deep networks in machine learning and AI, however, has inspired proposals for understanding how the brain might learn across multiple layers, and hence how it might implement or approximate BP. As of yet, none of these proposals have been rigorously evaluated on tasks where BP-guided deep learning has proved critical, or in architectures more structured than simple fully-connected networks. Here we present the first results on scaling up biologically motivated models of deep learning on datasets which need deep networks with appropriate architectures to achieve good performance. We present results on MNIST, CIFAR-10, and ImageNet and explore variants of the difference target-propagation (DTP) algorithm. We focus on DTP and introduce weight-transport-free variants modified to remove backpropagation from the penultimate layer, in both fully- and locally-connected architectures. These algorithms perform well for MNIST, but for CIFAR and ImageNet we find that DTP and variants perform significantly worse than BP, especially for network composed of locally connected units, opening questions about whether new architectures and algorithms are required to scale these approaches. Our results and implementation details help establish baselines for biologically motivated deep learning schemes going forward.