Hierarchical Representational Transformations of Working Memory in Brains and Machines

Qingqing Yang, Hyewon Willow Han, Bogeng Song, Julie Golomb, Dobromir Rahnev, Yalda Mohsenzadeh, Hsin-Hung Li

Keywords: working memory, representational geometry, cortical hierarchy, learning objectives, recurrent neural networks

TL;DR: This work shows working memory (WM) in humans and machines follows a shift from dynamic to stable representational transformation along the hierarchy, with gated recurrent dynamics and supervised learning best capturing human-like WM mechanisms.

Abstract: Working memory (WM) maintains past inputs while processing new ones, yet how representations transform between encoding and retrieval remains unclear. Clarifying whether these representations are sustained through stable coding formats, dynamically updated subspaces, or their interplay is key to uncovering the mechanisms of WM. To address this, we combined high-resolution 7T fMRI from the Natural Scenes Dataset with recurrent neural networks (RNNs) trained on a naturalistic 1-back task. Using representational similarity, cross-decoding, and subspace geometry analyses, we directly compared rotational and non-rotational transformations between WM encoding and retrieval phases in brain regions and model layers. Our analyses revealed convergent evidence for a mixture mechanism of WM coding for encoding and retrieval information: early visual regions (V1–hV4) underwent large representational changes across encoding to retrieval phases, including both rotational and non-rotational transformations. Whereas higher-order regions in the prefrontal cortex (FEF, dlPFC) were more stable. Applying the same analyses to models showed a similar mechanism across layers, but critically depended on the learning objective and the recurrent architecture. We examined two different encoder architectures, ResNet and Vision Transformer (ViT), each trained with supervised and self-supervised learning objectives. Models with supervised encoders preserved a hierarchical layer dissociation paralleling the cortical gradient in both rotational and non-rotational transformations, while models with self-supervised encoders diverged in the rotational transformation. Among recurrent architectures, gated architectures (GRU, LSTM) better reproduced the brain-like mixture of subspace rotational transformation. Taken together, these results established hierarchical shifts between flexibility and stability in WM representational transformation in both humans and machines, with supervised learning objectives combined with gated recurrent dynamics most closely resembling human WM mechanisms.

Primary Area: applications to neuroscience & cognitive science

Submission Number: 22834