An Information-Theoretic Approach to Diversity Evaluation of Prompt-based Generative Models

Mohammad Jalali, Azim Ospanov, Amin Gohari, Farzan Farnia

Keywords: Information measures, diversity evaluation, conditional generative models

Abstract: Text-conditioned generation models are commonly evaluated based on the quality of the generated data and its alignment with the input text prompt. On the other hand, several applications of prompt-based generative models require sufficient diversity in the generated data to ensure the models' capability of generating image and video samples possessing a variety of features. However, the existing diversity metrics are designed for unconditional generative models, and thus cannot distinguish the diversity arising from variations in text prompts and that contributed by the generative model itself. In this work, our goal is to quantify the prompt-induced and model-induced diversity in samples generated by prompt-based models. Specifically, we propose the application of matrix-based information measures to address this task, decomposing the kernel-based entropy $H(X)$ of generated data $X$ into the sum of conditional entropy $H(X|T)$, given text variable $T$, and mutual information $I(X; T)$. We show that this information-theoretic approach decomposes the existing Vendi diversity score defined based on $H(X)$ into the product of the following two terms: 1) Conditional-Vendi score based on $H(X|T)$ to quantify the model-induced diversity, and 2) Information-Vendi score based on $I(X; T)$ to measure the statistical relevance between $X$ and prompt $T$. Our theoretical results provide an interpretation for this diversity quantification and show that the Conditional-Vendi score aggregates the Vendi scores within the modes of a mixture prompt distribution. We conduct several numerical experiments to show the correlation between the Conditional-Vendi score and the internal diversity of text-conditioned generative models.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 10069