Efficient online learning with low-precision synaptic variables

Marcus K. Benna, Stefano Fusi

Abstract: Current neuromorphic devices suffer from major limitations in their ability to perform on-chip online learning. These limitations often derive from their poor memory capacity, which is due to the low precision of the variables representing the synaptic weights. Here we present simple constructions of synaptic models with low-precision dynamical variables that can continually store and preserve a large number of memories, which grows almost linearly with the number of synapses per neuron. In addition, the initial memory strength, which is related to the generalization ability of the network, is also high in these models, and scales approximately like the square root of the number of synapses. These favorable properties are obtained by orchestrating multiple interacting processes that operate on different timescales, to ensure the memory strength decays as slowly as the inverse square root of the age of the corresponding synaptic modification. This decay curve achieves an optimal compromise between large memory strengths and long lifetimes. We discuss digital implementations of such synapses suitable for neuromorphic hardware. They are efficient in the sense of requiring only a small number of bits per synapse, and respond robustly to auto-correlated sequences of synaptic modifications.