Residual Learning in Diffusion Models

Junyu Zhang, Daochang Liu, Eunbyung Park, Shichao Zhang, Chang Xu

Abstract: Diffusion models (DMs) have achieved remarkable generative performance, particularly with the introduction of stochastic differential equations (SDEs). Nevertheless, a gap emerges in the model sampling trajectory constructed by reverse-SDE due to the accumulation of score estimation and discretization errors. This gap results in a residual in the generated images, adversely impacting the image quality. To remedy this, we propose a novel residual learning framework built upon a correction function. The optimized function enables to improve image quality via rectifying the sampling trajectory effectively. Importantly, our framework exhibits transferable residual correction ability, i.e., a correctionfunction optimized for one pre-trained DM can also enhance the sampling trajectory constructed by other different DMs on the same dataset. Experimental results on four widely-used datasets demonstrate the effectiveness and transferable capability of our framework.