Continual Neural Computation

Matteo Tiezzi, Simone Marullo, Federico Becattini, Stefano Melacci

Abstract: Continuously processing a stream of not-i.i.d. data by neural models with the goal of progressively learning new skills is largely known to introduce significant challenges, frequently leading to catastrophic forgetting. In this paper we tackle this problem focusing on the low-level aspects of the neural computation model, differently from the most common existing approaches. We propose a novel neuron model, referred to as Continual Neural Unit (CNU), which does not only compute a response to an input pattern, but also diversifies computations to preserve what was previously learned, while being plastic enough to adapt to new knowledge. The values attached to weights are the outcome of a computational process which depends on the neuron input, implemented by a key-value map to select and blend multiple sets of learnable memory units. This computational mechanism implements a natural, learnable form of soft parameter isolation, virtually defining multiple computational paths within each neural unit. We show that such a computational scheme is related to the ones of popular models which perform computations relying on a set of samples stored in a memory buffer, including Kernel Machines and Transformers. Experiments in class-and-domain incremental streams processed in online and single-pass manner show how CNUs can mitigate forgetting without any replays or more informed learning criteria, while keeping competitive or better performance with respect to continual learning methods that explicitly store and replay data.