From sequences to schemas: How recurrent neural networks learn temporal abstractions

Vezha Boboeva, Alberto Pezzotta, George Dimitriadis, Athena Akrami

Keywords: statistical learning, abstract sequential pattern, generalisation, abstraction, classification, prediction, transfer learning, RNN, mechanistic interpretability

Abstract: A fundamental challenge in neuroscience is to understand how neural systems extract and represent abstract structure from complex, time-varying input. From language and music to action planning and sensory prediction, behavior relies on the ability to recognize relational patterns in sequences. Yet it remains unclear how such abstract temporal schemas are learned and encoded in neural population dynamics. Here, we show that training recurrent neural networks (RNNs) on an abstract sequence classification task drives the emergence of internal representations that express the underlying hierarchical structure and are supported by low-rank recurrent dynamics, whereas training on a standard next-token prediction task does not. Using sequences generated by a binary branching tree to instantiate abstract structure, we trained RNNs to classify sequences based on their abstract class (e.g., aab, aad → AAB; aba, aca → ABA), providing a label only at the end of the sequence. Despite the absence of explicit supervision at the transition level, the networks developed low-dimensional, linearly separable internal representations, reflecting the underlying hierarchical tree structure of the data, and encoding information about the sequence's path through this structure. This enables generalization across different token instantiations of the same abstract pattern. In contrast, RNNs trained on a next-token prediction task fail to form such organized dynamics or recover the underlying tree structure. However, through transfer learning, we show that when initialized with weights from a classification-trained network, prediction models learn faster and generalize better. These findings demonstrate that task objectives critically shape internal representations and that abstract structure, once learned, can serve as a reusable scaffold for diverse temporal computations.

Primary Area: applications to neuroscience & cognitive science

Submission Number: 24503