Sparse-Compression Diffusion Models

ICLR 2026 Conference Submission6110 Authors

Keywords: Sparse Coding, Diffusion Process, Representation Learning

Abstract: Diffusion models have demonstrated capabilities in generating synthetic data, especially the pictural ones. However, the conventional reverse diffusion process merely overfits in data denoising. Empirical studies indicate that the denoising mechanism obstructs the diffusion models to generate robust physical synthetic data. In this work, we propose the Sparse-Compression Diffusion Models (SCDM), with a novel reverse diffusion process that enforces compression. The compression attempts to recover the low-dimensional and sparse representations, potentially enabling the ability to capture the latent factors of some rules. SCDM demonstrates great generalization on datasets of images and physical scenarios. Our experiments also prove that the sparse latent representations learned by SCDM align well with scientific theories in physical scenarios.

Primary Area: generative models

Submission Number: 6110