Standardizing the Measurement of Text Diversity: A Tool and Comparative Analysis
Keywords: text diversity, summarization, generation
Abstract: The diversity across outputs generated by LLMs shapes perception of their quality and utility.
Achieving high textual diversity in datasets is often a desired quality, but there is no standard method to measure this aspect of model behaviour.
In this work we empirically investigate diversity scores on English texts and measure how much overlapping information is captured in these metrics.
We find that computationally efficient compression algorithms capture information similar to what is measured by slow-to-compute $n$-gram overlap homogeneity scores.
Further, a combination of measures---compression ratios, self-repetition of long $n$-grams and Self-BLEU and BERTScore---are sufficient to report, as they have low mutual correlation with each other.
The applicability of scores extends beyond analysis of generative models; for example, we highlight applications on instruction-tuning datasets and human-produced texts.
We release a diversity score package to facilitate research and invite consistency going forward.
Primary Area: foundation or frontier models, including LLMs
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 7107
Loading