Continuous Semi-Implicit Models

Longlin Yu, Jiajun Zha, Tong Yang, Tianyu Xie, Xiangyu Zhang, S.-H. Chan, Cheng Zhang

Abstract:

Semi-implicit distributions have shown great promise in variational inference and generative modeling.
Hierarchical semi-implicit models, which stack multiple semi-implicit layers, enhance the expressiveness of semi-implicit distributions and can be used to accelerate diffusion models given pretrained score networks.
However, their sequential training often suffers from slow convergence.
In this paper, we introduce CoSIM, a continuous semi-implicit model that extends hierarchical semi-implicit models into a continuous framework.
By incorporating a continuous transition kernel, CoSIM enables efficient, simulation-free training.
Furthermore, we show that CoSIM achieves consistency with a carefully designed transition kernel, offering a novel approach for multistep distillation of generative models at the distributional level.
Extensive experiments on image generation demonstrate that CoSIM performs on par or better than existing diffusion model acceleration methods, achieving superior performance on FD-DINOv2.

Lay Summary:

Multi-step generative models improve image quality and training efficiency, but distilling them into faster models still requires significant computational cost.
To address this, we propose CoSIM, a scalable training method that avoids slow step-by-step simulations. CoSIM unifies two advanced techniques — Hierarchical Semi-implicit Models for multi-step distillation and Score Identity Distillation for one-step distillation — within a continuous framework.

By incorporating a continuous transition kernel, CoSIM enables efficient, simulation-free training. Its training objective alternates between refining the image generator and updating an auxiliary score network, using a scaled Fisher divergence and an additional regularization term to improve multi-step generation quality.

In experiments on challenging image generation tasks, CoSIM matches or outperforms state-of-the-art diffusion model acceleration methods, achieving particularly strong results on FD-DINOv2. We hope this approach helps make high-quality generative models more efficient, scalable, and practical.

Primary Area: Deep Learning->Generative Models and Autoencoders

Keywords: Semi-Implicit Variational Inference, Score-Based Generative Models, Diffusion Models Distillation

Submission Number: 10594