Less is More: Adaptive Coverage for Synthetic Training Data

Sasan Tavakkol, Max Springer, Mohammadhossein Bateni, Neslihan Bulut, MohammadTaghi Hajiaghayi

Keywords: Large Language Models (LLMs), Synthetic Data Generation, Sampling Algorithms, Maximum Coverage Problem, Data Efficiency

TL;DR: Train a better classifier with less synthetic data from LLMs by intelligently selecting the most informative examples.

Abstract: Synthetic training data generation with Large Language Models (LLMs) like Google's Gemma and OpenAI's GPT offer a promising solution to the challenge of obtaining large, labeled datasets for training classifiers, especially when rapid model deployment is critical, such as classifying emerging social media trends or combating new forms of online abuse tied to current events. While prior research has examined the comparability of synthetic data to human-labeled data, this study introduces a novel sampling algorithm based on the maximum coverage problem to select a representative subset from a synthetically generated dataset. Our results demonstrate that training a classifier on this contextually sampled subset achieves superior performance compared to training on the entire dataset. This ``less is more'' approach not only improves accuracy but also reduces the volume of data required, leading to potentially more efficient training.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 9186