Go to ICLR 2024 Conference homepageA New Type of Associative Memory Network with Exponential Storage Capacity
Recent developments have sought to overcome the inherent limitations of traditional associative memory models, like Hopfield networks, where storage capacity scales linearly with input dimension. In this paper, we present a new extension of Hopfield networks that grants precise control over inter-neuron interactions while allowing control of the level of connectivity within the network. This versatile framework encompasses a variety of designs, including classical Hopfield networks, models with polynomial activation functions, and simplicial Hopfield networks as particular cases. Remarkably, a specific instance of our construction, resulting in a new self-attention mechanism, is characterized by quasi-exponential storage capacity and a sparse network structure, aligning with biological plausibility. To our knowledge, our proposed construction introduces the first biologically-plausible associative memory model with exponential storage capacity. Furthermore, the resulting model admits a very efficient implementation via vectorization; therefore, it can fully exploit modern numerical computation hardware like GPUs. This work not only advances the theoretical foundations of associative memory but also provides insights into the development of neurobiologically inspired associative memory systems with unprecedented capabilities.