Variance reduction of diffusion model's gradients with Taylor approximation-based control variate

Paul Jeha, Will Sussman Grathwohl, Michael Riis Andersen, Carl Henrik Ek, Jes Frellsen

Keywords: Variance reduction, score based model, generative model, diffusion model

TL;DR: A Taylor expansion based control variate to reduce the variance in the gradients of a diffusion model

Abstract: Score-based models, trained with denoising score matching, are remarkably effective in generating high dimensional data. However, the high variance of their training objective hinders optimisation. We attempt to reduce it with a control variate, derived via a $k$-th order Taylor expansion on the training objective and its gradient. We prove an equivalence between the two and demonstrate empirically the effectiveness of our approach on a low dimensional problem setting; and study its effect on larger problems.

Submission Number: 61