Low-Rank Successor Representations Capture Human-Like Generalization

Eva Yi Xie, Nathaniel D. Daw, Benjamin Eysenbach

Track: Extended Abstract Track

Keywords: successor representation, contrastive reinforcement learning, low-rank predictive representations

TL;DR: Low-rank successor representations enhance task performance and align with human behavior, in comparison to its tabular form. This suggests low-rank predictive representation as a potentially shared learning principle across brains and machines.

Abstract:

Intelligent behavior is hypothesized to hinge on predictive maps of future states. The successor representation (SR) formalizes such maps and has been influential in reinforcement learning and cognitive neuroscience.
The SR is typically represented in a tabular form, where each entry stores the probability of transitioning from one state to another, but this approach does not generalize to states unseen during training.
We thus study the function approximation of the SR, asking whether it yields more efficient solutions to navigation tasks and captures human-like behavioral patterns. Potentially, the same representational structure that supports efficient navigation may also shape systematic biases in behavior. To learn such approximations, we use a contrastive reinforcement learning objective.
In gridworlds, we find that function-approximated SRs generalize to novel states and produce smoother representations; this includes successfully navigating out-of-distribution state-goal pairs and generating shorter paths on tasks solvable by tabular SRs.
In high-dimensional graph learning benchmarks, the function-approximated SRs are low-rank and resemble human behavioral patterns across graph topologies, whereas tabular SRs would lack such inductive bias.
Taken together, our results position function-approximated SRs as a practical framework for probing the representational structure that underlies flexible and efficient planning in both biological and artificial systems.

Submission Number: 74