Hyperbolic Associative Memory Networks

Boliang Hao, Bailing Zhang, Fangyu Wu

Keywords: Hyperbolic Geometry, Modern Hopfield Networks, Associative Memory, Riemannian Optimization, Hierarchical Representation Learning

TL;DR: We extend modern Hopfield networks to hyperbolic space with a model-agnostic, Riemannian energy formulation, yielding a plug-and-play memory module that excels on deeply hierarchical data while matching Euclidean baselines on shallow cases.

Abstract:

Despite the widespread success of associative memory models, such as modern Hopfield networks, in various domains, related research has still been confined to Euclidean or kernel-induced spaces. When the state space is restricted to Euclidean geometry, it becomes challenging to accurately capture hierarchical structures in the data. For instance, even in high-dimensional Euclidean space, arbitrary tree structures cannot be embedded with minimal distortion; in many tasks requiring the processing of hierarchical data, Hopfield networks based on Euclidean representations tend to introduce bias and distortion in semantic relations. To address this issue, we propose extending modern Hopfield retrieval to hyperbolic space. Specifically, we map query and memory vectors from Euclidean space to hyperbolic space via exponential maps and define an energy function based on the Minkowski inner product, with a solid theoretical foundation. The retrieval process uses Riemannian manifold optimization, combining curvature-aware gradients with exponential maps to ensure that the optimization trajectory remains on the manifold and produces stable updates.
Our central view can be stated as a hierarchy-sensitivity hypothesis: when the data exhibit clear and deeper hierarchical structure, hyperbolic geometry brings statistically significant improvements; when the hierarchy is weak or only shallow, performance shows no significant difference from Euclidean modern Hopfield networks. We validate this through depth-controlled comparisons and cross-level consistency metrics, and the empirical results are consistent with the hypothesis.
Accordingly, the proposed hyperbolic associative memory can serve as a plug-and-play general memory module embedded into task architectures that require hierarchical understanding, for storing and retrieving raw inputs, intermediate representations, or learned prototypes, and explicitly exploiting hierarchical information.
Our method is formulated in a model-agnostic manner and applies to any hyperbolic model with constant negative curvature.
We instantiate it with the Poincaré ball for experiments.

Primary Area: learning on graphs and other geometries & topologies

Submission Number: 22346