Does Instruction Tuning Reduce Diversity? A Case Study Using Code Generation

Alexander Shypula, Shuo Li, Botong Zhang, Vishakh Padmakumar, Kayo Yin, Osbert Bastani

Keywords: LLM, Diversity, RLHF, DPO, SFT, Program Synthesis, Evaluation

TL;DR: We find Preference Tuning (i.e. DPO/PPO) increases diversity in content, but it reduces the syntactic diversity and the N-Gram diversity of generations using a novel dataset for measuring diversity with programs.

Abstract: Large Language Models (LLMs) should ideally generate diverse content for open-ended prompts (e.g., variety in cooking recipes). Anecdotal evidence has suggested that preference-tuned language models struggle to generate diverse content, which would have important implications for how we align models. However, research on this question has been limited by the difficulty of measuring diversity, which naively would require costly human evaluation. We propose to leverage code as a means to study semantic diversity, since code has executable semantics. To this end, we create an open-ended program synthesis task, enabling us to cheaply evaluate the diversity of hundreds of thousands of generations. Using our methodology, we find that while instruction-tuning reduces syntactic and lexical diversity, it can actually increase semantic diversity. We also study the effect of model size and prompting technique on diversity. Finally, we find that neural diversity metrics correlate poorly with our semantic diversity metrics, highlighting the need for more rigorous methodologies for evaluating diversity.

Primary Area: interpretability and explainable AI

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 12427