DISCO: Diversifying Sample Condensation for Efficient Model Evaluation

Alexander Rubinstein, Benjamin Raible, Martin Gubri, Seong Joon Oh

Keywords: efficient evaluation, early drift detection, frequent evaluation, reliable monitoring

Abstract: To catch or avoid drift in deployed and in-development models, we need to evaluate them as often as possible. For modern models, a single training run can produce hundreds of checkpoints, e.g., OLMo 2-32B has 753 on Hugging Face. Evaluating each of these checkpoints is currently impossible because even a single full evaluation becomes prohibitively expensive. Benchmarks such as LMMs-Eval and HELM demand thousands of GPU hours per model. To address the growing cost of standard evaluation, new methods focused on efficient evaluation have started to appear. The typical approach follows two steps. First, select an anchor subset of data. Second, train a mapping from the accuracy on this subset to the final test result. The drawback is that anchor selection depends on clustering, which can be complex and sensitive to design choices. We argue that promoting diversity among samples is not essential; what matters is to select samples that maximise diversity in model responses. Our method, Diversifying Sample Condensation (DISCO), selects the top-k samples with the greatest model disagreements. This uses greedy, sample-wise statistics rather than global clustering. The approach is conceptually simpler. From a theoretical view, inter-model disagreement provides an information-theoretically optimal rule for such greedy selection. DISCO shows empirical gains over prior methods, achieving state-of-the-art results in performance prediction across MMLU, Hellaswag, Winogrande, and ARC. By enabling cheap, repeatable performance estimation, DISCO supports operating at scale: reliable monitoring and comparison of many models with minimal compute, so that frequent checkpoint evaluation and drift detection become feasible during development and in deployed ML systems.

Submission Number: 39