Distributional Autoencoders Know the Score

Andrej Leban

Keywords: Generative Models, Unsupervised Learning Methods, Autoencoders, Nonlinear Dimensionality Reduction and Manifold Learning, Score Matching / Score-based Models, Scientific Machine Learning / AI for Science

TL;DR: For Distributional Principal Autoencoders we prove a closed-form data-score–geometry identity and that any latent coordinates beyond the data manifold are uninformative.

Abstract: The Distributional Principal Autoencoder (DPA) combines distributionally correct reconstruction with principal-component-like interpretability of the encodings. In this work, we provide exact theoretical guarantees on both fronts. First, we derive a closed-form relation linking each optimal level-set geometry to the data-distribution score. This result explains DPA's empirical ability to disentangle factors of variation of the data, as well as allows the score to be recovered directly from samples. When the data follows the Boltzmann distribution, we demonstrate that this relation yields an approximation of the minimum free-energy path for the Müller–Brown potential in a single fit. Second, we prove that if the data lies on a manifold that can be approximated by the encoder, latent components beyond the manifold dimension are conditionally independent of the data distribution - carrying no additional information - and thus reveal the intrinsic dimension. Together, these results show that a single model can learn the data distribution and its intrinsic dimension with exact guarantees simultaneously, unifying two longstanding goals of unsupervised learning.

Primary Area: Deep learning (e.g., architectures, generative models, optimization for deep networks, foundation models, LLMs)

Submission Number: 24755