Learning sequence models through consolidation
Keywords: neuroscience, large language models, consolidation
TL;DR: We present a model in which sequences encoded in the hippocampus are replayed to train a generative network (simulated by GPT-2) to capture the transition statistics of memories.
Abstract: Episodic memory is a reconstructive process, thought to depend on schema-based predictions made by generative models learned through systems consolidation. We extend previous work on memory for static scenes to model the construction and consolidation of sequential experience. After sequences are encoded in the hippocampus, a network is trained to predict the next item in a sequence during replay (simulated by training GPT-2 on a range of stimuli). The resulting model can memorise narratives, with characteristic gist-based distortions, and can also be applied to non-linguistic tasks such as spatial and relational inference. In addition, we explore `retrieval augmented generation', in which sequence generation is conditioned on relevant ‘memories’, as a model for how hippocampal specifics can be combined with neocortical general knowledge.
Submission Number: 18
Loading