Identifying Learning Rules from Neural Network Observables

Aran Nayebi, Sanjana Srivastava, Surya Ganguli, Daniel LK Yamins

Abstract:

The brain modifies its synaptic strengths during learning in order to better adapt to
its environment. However, the underlying plasticity rules that govern learning are
unknown. Many proposals have been suggested, including Hebbian mechanisms,
explicit error backpropagation, and a variety of alternatives. It is an open question
as to what specific experimental measurements would need to be made to determine whether any given learning rule is operative in a real biological system. In
this work, we take a “virtual experimental” approach to this problem. Simulating
idealized neuroscience experiments with artificial neural networks, we generate
a large-scale dataset of learning trajectories of aggregate statistics measured in
a variety of neural network architectures, loss functions, learning rule hyperparameters, and parameter initializations. We then take a discriminative approach,
training linear and simple non-linear classifiers to identify learning rules from
features based on these observables. We show that different classes of learning
rules can be separated solely on the basis of aggregate statistics of the weights,
activations, or instantaneous layer-wise activity changes, and that these results
generalize to limited access to the trajectory and held-out architectures and learning
curricula. We identify the statistics of each observable that are most relevant for
rule identification, finding that statistics from network activities across training are
more robust to unit undersampling and measurement noise than those obtained
from the synaptic strengths. Our results suggest that activation patterns, available
from electrophysiological recordings of post-synaptic activities on the order of
several hundred units, frequently measured at wider intervals over the course of
learning, may provide a good basis on which to identify learning rules.