Computational Model of the Hippocampus Drives Exploratory Behaviour in Reinforcement Learning Agents

Sabrina Du, Adel Halawa, Aleksei Efremov, Daniel Levenstein, Blake Aaron Richards

Abstract: The hippocampus supports spatial navigation, memory, and planning through the formation of a cognitive map: a structured environmental representation reflected in its neural activity. These neural dynamics, and the behaviors that arise from them, can be modelled computationally using artificial neural networks (ANNs). However, these models typically leverage reinforcement learning (RL) using external rewards, failing to capture the intrinsic drive of freely exploring animals in the absence of external rewards. This study aims to investigate whether a recurrent neural network (RNN) exhibiting hippocampal-like activity can drive exploratory behavior in reward-free RL agents. We leveraged an existing RNN trained for sensory sequence prediction, exhibiting hippocampal-like activity patterns, and used its prediction error as the intrinsic reward to train an Actor-Critic agent. In a Novel Object Recognition task (NOR), the RL agent occupied the region of interest significantly more than a random control (peak at 37.2% ± 1.8% vs. 15.0% ± 1.2% [mean ± SEM], Welch's t-test, p < 0.01). This preference persisted in a multi-room environment, where the agent spent 8.2% ± 0.2% of its time in the most distant, novel room, compared to 0.023% ± 0.017% of the random control (p < 0.001). These results demonstrate that cognitive maps derived from sensory prediction can drive exploration and novelty-seeking, providing a computational framework for how hippocampal dynamics guide navigation in the absence of external incentives.