Low Tensor Rank Learning of Neural Dynamics
Keywords: Low-rank, tensors, learning, recurrent neural networks, dynamical systems
TL;DR: The tensor formed by stacking weight matrices over training is low rank.
Abstract: Recent work has shown that the weight matrices of task-trained RNNs are typically low rank, but how this low rank structure unfolds over learning is unknown. To address this, we investigate the rank of the 3-tensor formed by the weight matrices throughout learning. By fitting RNNs of varying rank to large-scale neural recordings during learning, we find that the inferred weights are low-tensor-rank and therefore evolve over a fixed low-dimensional subspace throughout the entire course of learning. We next validate the observation of low-tensor-rank learning on RNNs trained to solve various tasks. Finally, we present a set of mathematical results bounding the matrix and tensor ranks of gradient descent learning dynamics which show that low-tensor-rank weights emerge naturally in RNNs trained to solve low-dimensional tasks. Taken together, our findings provide insight on the evolution of connectivity over learning in both biological and artificial neural networks.
Submission Number: 14
Loading