Binary associative memory networks: A review of mathematical framework and capacity analysis

HAN BAO, Zhongying Zhao

Abstract:

In recent years, heightened interest has been ignited in associative memory networks, largely
attributed to their perceived equivalence with the attention mechanism, a fundamental component
of the Transformer architecture. The opaque nature of deep neural networks, often characterized
as “black boxes”, has intensified the pursuit of explainability, positioning associative memory
networks as promising candidates for illuminating the inherent complexities of deep learning
models. Despite their increasing prominence, the mathematical analysis of their capacity remains
a significant research gap, which constitutes the central focus of this paper. To address this
gap, we commence with a review of the mathematical framework underpinning associative
memory networks, with particular emphasis on their binary configurations, drawing insights from
the derivation of the dense associative memory model. Additionally, we review a systematic
methodology for analyzing the capacity of binary associative memory networks, building upon
established studies of dense associative memory networks. Utilizing this analytical framework, we
derive the capacity of several prominent associative memory networks, including binary modern
Hopfield networks and binary spherical Hopfield networks. Through comprehensive discussions
and rigorous deductions, we aim to elucidate the characteristics of binary associative memory
networks, thereby providing valuable insights and practical guidance for their effective application
in real-world scenarios.