Holdout-Loss-Based Data Selection for LLM Finetuning via In-Context Learning

Ling Zhang, Xianliang Yang, Juwon Yu, Park Cheonyoung, Lei Song, Jiang Bian

Keywords: LLM Fine-tuning (SFT, DPO, SimPO); Data selection; Holdout loss; In-context learning; Gradient reweighting

TL;DR: We present a holdout-loss-based data selection framework that leverages in-context learning for efficient computation.

Abstract:

Fine-tuning large pretrained language models is a common approach for aligning them with human preferences, but noisy or off-target examples can dilute supervision. While small, well-chosen datasets often match the performance of much larger ones, systematic and efficient ways to identify high-value training data remain underexplored. Many current methods rely on heuristics or expensive retraining. We present a principled, resource-efficient framework for data selection and reweighting. At its core is an In-Context Approximation (ICA) that estimates the holdout loss a model would incur after training on a candidate example by conditioning on a small, curated holdout set in context. ICA requires no reference model and no additional finetuning.
We define the resulting estimate as the ICA score, and derive per-example weights that dynamically reweight gradient updates as model parameters evolve. Across SFT, DPO, and SimPO, and over diverse backbones and datasets, ICA-based reweighting consistently improves model alignment with minimal overhead. We analyze sensitivity to score update frequency and the number of in-context holdout examples. We also discuss
limitations in rapidly drifting on-policy settings, highlighting directions for future work.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 8899