Predictive Attractor Models

Ramy Mounir, Sudeep Sarkar

Keywords: Sequential Memory, Predictive Models, Fixed-point Attractors, Associative Memory Models, State Space Models, Biologically plausible, Hebbian Plasticity Rules, Local Computations, Hierarchical Temporal Memory, Continual Learning, Multiple Possibilities Generation, Noise Tolerance

TL;DR: A biologically plausible sequential memory model that (1) represents and generates multiple possibilities by learning fixed-point attractors, (2) learns continually while avoiding catastrophic forgetting, and (3) is robust to noise.

Abstract: Sequential memory, the ability to form and accurately recall a sequence of events or stimuli in the correct order, is a fundamental prerequisite for biological and artificial intelligence as it underpins numerous cognitive functions (e.g., language comprehension, planning, episodic memory formation, etc.) However, existing methods of sequential memory suffer from catastrophic forgetting, limited capacity, slow iterative learning procedures, low-order Markov memory, and, most importantly, the inability to represent and generate multiple valid future possibilities stemming from the same context. Inspired by biologically plausible neuroscience theories of cognition, we propose Predictive Attractor Models (PAM), a novel sequence memory architecture with desirable generative properties. PAM is a streaming model that learns a sequence in an online, continuous manner by observing each input only once. Additionally, we find that PAM avoids catastrophic forgetting by uniquely representing past context through lateral inhibition in cortical minicolumns, which prevents new memories from overwriting previously learned knowledge. PAM generates future predictions by sampling from a union set of predicted possibilities; this generative ability is realized through an attractor model trained alongside the predictor. We show that PAM is trained with local computations through Hebbian plasticity rules in a biologically plausible framework. Other desirable traits (e.g., noise tolerance, CPU-based learning, capacity scaling) are discussed throughout the paper. Our findings suggest that PAM represents a significant step forward in the pursuit of biologically plausible and computationally efficient sequential memory models, with broad implications for cognitive science and artificial intelligence research. Illustration videos and code are available on our project page: https://ramymounir.com/publications/pam.

Primary Area: Neuroscience and cognitive science (neural coding, brain-computer interfaces)

Submission Number: 1551