Building, Reusing, and Generalizing Abstract Representations from Concrete Sequences
Keywords: Abstraction, Chunking, Cognitive Science, LLMs
TL;DR: The paper presents a hierarchical variable learning model (HVM) that efficiently abstracts patterns in sequences, outperforming standard compression methods and large language models in mimicking human memory and generalization.
Abstract: Humans excel at learning abstract patterns across different sequences, filtering out
irrelevant details, and transferring these generalized concepts to new sequences.
In contrast, many sequence learning models lack the ability to abstract, which
leads to memory inefficiency and poor transfer. We introduce a non-parametric
hierarchical variable learning model (HVM) that learns chunks from sequences
and abstracts contextually similar chunks as variables. HVM efficiently organizes
memory while uncovering abstractions, leading to compact sequence representations.
When learning on language datasets such as babyLM, HVM learns a more efficient
dictionary than standard compression algorithms such as Lempel-Ziv. In a sequence
recall task requiring the acquisition and transfer of variables embedded in sequences,
we demonstrate HVM’s sequence likelihood correlates with human recall times. In
contrast, large language models (LLMs) struggle to transfer abstract variables as
effectively as humans. From HVM’s adjustable layer of abstraction, we demonstrate
that the model realizes a precise trade-off between compression and generalization.
Our work offers a cognitive model that captures the learning and transfer of abstract
representations in human cognition and differentiates itself from LLMs.
Primary Area: transfer learning, meta learning, and lifelong learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 6911
Loading