A Bias-Variance-Covariance Decomposition of Kernel Scores for Generative Models

Sebastian Gregor Gruber, Florian Buettner

Primary Area: learning theory

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Kernel Score, Generative Models, Bias-Variance-Covariance Decomposition, Image Generation, Audio Generation, Natural Language Generation, Diffusion Models, Large Language Models

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: This paper introduces a framework for estimating uncertainty in generative models, which can be applied to various domains like image, audio, and language generation and outperforms existing methods on question answering datasets.

Abstract:

Generative models, like large language models, are becoming increasingly relevant in our daily lives, yet a theoretical framework to assess their generalization behavior and uncertainty does not exist.
Particularly, the problem of uncertainty estimation is commonly solved in an ad-hoc manner and task-dependent.
For example, natural language approaches cannot be transferred to image generation.
In this paper, we introduce the first bias-variance-covariance decomposition for kernel scores and their associated entropy.
We propose unbiased and consistent estimators for each quantity which only require generated samples but not the underlying model itself.
As an application, we offer a generalization evaluation of diffusion models and discover how mode collapse of minority groups is a contrary phenomenon to overfitting.
Further, we demonstrate that variance and predictive kernel entropy are viable measures of uncertainty for image, audio, and language generation.
Specifically, our approach for uncertainty estimation is more predictive of performance on CoQA and TriviaQA question answering datasets than existing baselines and can also be applied to closed-source models.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3423