Statistical Inference for Generative Model Comparison

Zijun Gao, Han Su, Yan Sun

Abstract: Generative models have achieved remarkable success across a range of applications, yet their evaluation still lacks principled uncertainty quantification. In this paper, we develop a method for comparing how close different generative models are to the underlying distribution of test samples. Particularly, our approach employs the Kullback-Leibler (KL) divergence to measure the distance between a generative model and the unknown test distribution, as KL requires no tuning parameters such as the kernels used by RKHS-based distances. And the relative KL divergence is the only $f$-divergence that admits a crucial cancellation of the hard-to-estimate term to enable the faithful uncertainty quantification. Furthermore, we extend our method to comparing conditional generative models and leverage Edgeworth expansions to address limited-data settings. On simulated datasets with known ground truth, we show that our approach realizes effective coverage rates, and has higher power compared to kernel-based methods. When applied to generative models on image and text datasets, our procedure yields conclusions consistent with benchmark metrics but with statistical confidence. The source code to reproduce our experiments is available at https://github.com/sylydya/compare-generative-models.

Certifications: J2C Certification

Submission Type: Regular submission (no more than 12 pages of main content)

Code: https://github.com/sylydya/compare-generative-models

Assigned Action Editor: ~Jes_Frellsen1

Submission Number: 6631