Seeds of Structure: Patch PCA Reveals Universal Compositional Cues in Diffusion Models

Qingsong Wang, Zhengchao Wan, Mikhail Belkin, Yusu Wang

Keywords: Diffusion models, Principal Component Analysis, Generative model

TL;DR: Diffusion noise seeds contain universal compositional blueprints that persist across datasets, enabling our Patch PCA framework for zero-shot structure control.

Abstract: Diffusion models transform random noise into images of remarkable fidelity, yet the structure of this noise-to-image map remains largely unexplored. We investigate this relationship using patch-wise Principal Component Analysis (PCA) and empirically demonstrate that low-frequency components of the initial noise predominantly influence the compositional structure of generated images. Our analyses reveal that noise seeds inherently contain universal compositional cues, evident when identical seeds produce images with similar structural attributes across different datasets and model architectures. Leveraging these insights, we develop and theoretically justify a simple yet effective Patch PCA denoiser that extracts underlying structure from noise using only generic natural image statistics. The robustness of these structural cues is observed to persist across both pixel-space models and latent diffusion models, highlighting their fundamental nature. Finally, we introduce a zero-shot editing method that enables injecting compositional control over generated images, providing an intuitive approach to guided generation without requiring model fine-tuning or additional training.

Primary Area: Deep learning (e.g., architectures, generative models, optimization for deep networks, foundation models, LLMs)

Submission Number: 24796