Maximum Entropy under Carre du Champ Constraints

Tanya Veeravalli, Atsushi Nitanda

Keywords: maximum entropy, generative models, memorization, variational principle, geometric complexity

TL;DR: We propose a new way to do maximum entropy exploration in an unknown lower-dimensional data manifold in a way that mitigates memorization of training data samples, and propose a geometric variational principle.

Abstract: Recent works have shown generative models' striking capabilities to memorize training data, and encode the underlying lower-dimensional data manifold $M$. However, while this phenomenon has been studied from different perspectives, current training objectives do not explicitly \textit{control} the geometric complexity of memorization. In this work, we propose a constrained maximum-entropy (MaxEnt) principle for learning a generative model density $q$ that (i) fits the data, (ii) discourages tangential ``memorization'' (training-point attractors along the manifold) through a data-driven Carr\'{e}-du-Champ (CDC) tangential Fisher constraint defined on the projected marginal of $q$ onto $M$, and (iii) enforces bounded normal thickness so the ambient entropy objective is well-posed. Leveraging this, we prove KL-to-uniform control and explicit anti-concentration bounds near training points, and lift them to ambient space via thickness control. We also provide scalable kNN/local-PCA estimators for the data-driven geometric terms and demonstrate numerical and simulation results. More generally, to our knowledge, this is the first work proposing a principled variational principle that jointly controls entropy and geometric complexity through a data-dependent metric.

Submission Number: 38