Score-based Conditional Generation with Fewer Labeled Data by Self-calibrating Classifier Guidance

Paul Kuo-Ming Huang, Si-An Chen, Hsuan-Tien Lin

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Score-based Model, Classifier Guidance, Conditional Generation

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: Improve classifier-guided score-based conditional generation through regularizing the classifier during training time.

Abstract: Score-based generative models (SGMs) are a popular family of deep generative models that achieve leading image generation quality. Early studies extend SGMs to tackle class-conditional generation by coupling an unconditional SGM with the guidance of a trained classifier. Nevertheless, such classifier-guided SGMs do not always achieve accurate conditional generation, especially when trained with fewer labeled data. We argue that the problem is rooted in the classifier's tendency to overfit without coordinating with the underlying unconditional distribution. We propose improving classifier-guided SGMs by letting the classifier regularize itself to respect the unconditional distribution. Our key idea is to use principles from energy-based models to convert the classifier as another view of the unconditional SGM. Then, existing loss for the unconditional SGM can be leveraged to achieve regularization by calibrating the classifier's internal unconditional scores. The regularization scheme can be applied to not only the labeled data but also unlabeled ones to further improve the classifier. Empirical results show that the proposed approach significantly improves conditional generation quality across various percentages of fewer labeled data. The results confirm the potential of the proposed approach for generative modeling with limited labeled data.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3038