Statistics-guided Associative Memories

Hongzhi Wang, Satyananda Kashyap, Niharika Shimona D'Souza, Tanveer Syeda-mahmood

Keywords: Associative memories, joint statistics

TL;DR: Statistics-guided Associative Memories

Abstract: Content-associative memories such as Hopfield networks have been studied as a good mathematical model of the auto-associative features in the CA3 region of the hippocampal memory system. Modern Hopfield networks (MHN) are generalizations of the classical Hopfield networks with revised energy functions and update rules to expand storage to exponential capacity. However, they are not yet practical due to spurious metastable states leading to recovery errors during memory recall. In this work, we present a fresh perspective on associative memories using joint co-occurrence statistics, and show that accurate recovery of patterns is possible from a partially-specified query using the maximum likelihood principle. In our formulation, memory retrieval is addressed via estimating the joint conditional probability of the retrieved information given the observed associative information. Unlike previous models that have considered independence of features, we do recovery under the maximal dependency assumption to obtain an upper bound on the joint probability of occurrence of features. We show that this new approximation substantially improves associative memory retrieval accuracy on popular benchmark datasets.

Submission Number: 18