Dense Hopfield Networks with Hierarchical Memories

Aditya Cowsik, Adithya Sriram

Track: long paper (up to 5 pages)

Keywords: Hopfield, Generalization, Hierarchical data

TL;DR: We exactly solve for some dynamics in a dense hopfield model with hierarchically generated data and find polynomial data requirements with respect to a data complexity measure.

Abstract: We consider a 3-level hierarchical generative model for memories which are sampled and stored in a dense Hopfield network with polynomial activation. We analytically derive conditions for each level of this hierarchy to be locally stable -- that is they are local energy maxima. We find that it takes only a polynomial amount of information to generalize beyond particular memories and even particular groups in the hierarchy. Our theory predicts the qualitative features a phase diagram in the number of memories, sharpness of the activation function (polynomial degree) for data from Fashion-MNIST.

Submission Number: 21