On low-dimensional representations and associative memory energy functions

David Lipshutz

Keywords: Associative memory models; representation learning; low-dimensional structure

TL;DR: Minimizing the associative memory model energy function is equivalent to minimizing the dimensionality of the model’s representation.

Abstract: Associative memory (AM) models encode memories as local minima of energy functions and are generally analyzed in terms of their storage capacity. We provide a complementary perspective: minimizing the AM energy function is related to minimizing the dimensionality of the model’s representation. Starting from a representational objective that balances input reconstruction with maximization of the inverse participation ratio (IPR), we relate the objective to minimization of the classical AM energy function. We further generalize the relation to dense AM energy functions using kernel methods.

Submission Number: 2