Emergence of Spatial Representation in an Actor-Critic Agent with Hippocampus-Inspired Sequence Generator

ICLR 2026 Conference Submission22798 Authors

Keywords: neuroscience, deep reinforcement learning, place cells, navigation

TL;DR: A minimal, sparsely driven sequence generator in actor-critic agent not only supports successful navigation but also gives rise to hippocampus-like spatial representations.

Abstract:

Sequential activation of place-tuned neurons in an animal during navigation is typically interpreted as reflecting the sequence of input from adjacent positions along the trajectory. More recent theories about such place cells suggest sequences to arise from abstract cognitive objectives like planning. Here, we propose a mechanistic and parsimonious interpretation to complement these ideas: hippocampal sequences arise from intrinsic recurrent circuitry that propagates activity without sustained input, acting as a temporal memory buffer for extremely sparse inputs.
We implement a minimal sequence generator inspired by neurobiology and pair it with an actor–critic learner for egocentric visual navigation. Our agent reliably solves a continuous maze without explicit geometric cues, with performance depending on the length of the recurrent sequence. Crucially, the model outperforms LSTM cores under sparse input conditions (16 channels, $\sim2.5$% activity), but not under dense input, revealing a strong interaction between representational sparsity and memory architecture.
Hidden units develop localized place fields, distance-dependent spatial kernels, and task-dependent remapping, while inputs to the generator orthogonalize and spatial information increases across layers. These phenomena align with neurobiological data and are causal to performance. Together, our results show that sparse input synergizes with sequence-generating dynamics, providing both a mechanistic account of place cell sequences in the mammalian hippocampus and a simple inductive bias for reinforcement learning based on sparse egocentric inputs in navigation tasks.

Primary Area: applications to neuroscience & cognitive science

Submission Number: 22798