Unraveling the Latent Hierarchical Structure of Language and Images via Diffusion Models

Antonio Sclocchi, Noam Itzhak Levi, Alessandro Favero, Matthieu Wyart

Keywords: data structure, hierarchical compositionality, diffusion models, statistical physics

TL;DR: Hierarchical compositionality in data structure predicts a diverging length-scale at a phase transition observed in diffusion models.

Abstract: High-dimensional data must be highly structured to be learnable. Although the compositional and hierarchical nature of data is often put forward to explain learnability, quantitative measurements establishing these properties are scarce. Likewise, accessing the latent variables underlying such a data structure remains a challenge. Forward-backward experiments in diffusion-based models, where a datum is noised and then denoised, are a promising tool to achieve these goals. We predict in simple hierarchical models that in this process, changes in data occur by correlated chunks, with a length scale that diverges at a noise level where a phase transition is known to take place. Remarkably, we confirm this prediction in both text and image datasets using state-of-the-art diffusion models. Our results suggest that forward-backward experiments are informative on the nature of latent variables, and that the effect of changing deeper ones is revealed near the transition.

Submission Number: 34