## Maximum Likelihood Training of Score-Based Diffusion Models

21 May 2021, 20:49 (modified: 24 Oct 2021, 17:38)NeurIPS 2021 SpotlightReaders: Everyone
Keywords: generative models, density estimation, score matching, score-based generative models, diffusion models, stochastic differential equations, normalizing flows, neural ODEs, likelihood, continuous normalizing flows
TL;DR: Score-based generative models can achieve state-of-the-art likelihoods when re-weighting the training objective.
Abstract: Score-based diffusion models synthesize samples by reversing a stochastic process that diffuses data to noise, and are trained by minimizing a weighted combination of score matching losses. The log-likelihood of score-based diffusion models can be tractably computed through a connection to continuous normalizing flows, but log-likelihood is not directly optimized by the weighted combination of score matching losses. We show that for a specific weighting scheme, the objective upper bounds the negative log-likelihood, thus enabling approximate maximum likelihood training of score-based diffusion models. We empirically observe that maximum likelihood training consistently improves the likelihood of score-based diffusion models across multiple datasets, stochastic processes, and model architectures. Our best models achieve negative log-likelihoods of 2.83 and 3.76 bits/dim on CIFAR-10 and ImageNet $32\times 32$ without any data augmentation, on a par with state-of-the-art autoregressive models on these tasks.
Supplementary Material: pdf
Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.
Code: https://github.com/yang-song/score_flow
14 Replies