Evaluating Sample Utility for Efficient Data Selection by Mimicking Model Weights

Tzu-Heng Huang, Manjot Bilkhu, John Cooper, Frederic Sala, Javier Movellan

TL;DR: We propose an efficient, model-based data selection framework that leverages weight-space geometry to prioritize samples to learn, estimate overall sample utility, and create effective data filters.

Abstract: Large-scale web-crawled datasets contain noise, bias, and irrelevant information, necessitating data selection techniques. Existing methods depend on hand-crafted heuristics, downstream datasets, or require expensive influence-based computations---all of which limit scalability and introduce unwanted data dependencies. To address this, we introduce the Mimic Score, a simple and geometry-based data-quality metric that evaluates utility by measuring alignment between a sample’s gradients and a target direction induced by a pre-trained reference model. This leverages readily available model weights, avoids needing validation datasets, and incurs minimal computational overheads. Building on this metric, we propose Grad-Mimic, a two-stage framework that re-weights samples online to accelerate training and aggregates sample utilities offline to construct effective data filters. Empirically, we show that using mimic scores to guide training improves data efficiency, accelerates convergence, yields consistent performance gains across six image datasets, and enhances CLIP models with 20.7\% fewer training steps. Additionally, mimic score-based filters augment existing filtering techniques, enabling improved CLIP models trained with 4.7 million fewer samples.

Lay Summary:

Modern models rely on massive web-crawled datasets full of noise, yet existing data selection methods depend on hand-crafted heuristics, curated validation sets, or expensive influence computations---all limiting scalability and adding overhead. We propose a simpler, dataset-agnostic, compute-efficient way to identify high-value data.

We introduce the Mimic Score, a geometry-based metric that measures how well a sample's gradient aligns with a direction pointing toward a pre-trained reference model's weights. This exploits today's access asymmetry---powerful models are public while their training data stays proprietary---and avoids validation data or extra inference. Building on it, we develop Grad-Mimic, a two-stage framework that re-weights samples online during training and aggregates their utilities offline to build effective data filters.

Grad-Mimic improves data efficiency across six image datasets, trains CLIP models with 20.7% fewer steps, and builds filters that remove 4.7 million low-value samples while boosting performance---all with minimal overhead. We show that publicly released models can serve as reliable reference models for computing Mimic Scores. These scores accurately identify noisy samples and estimate training dataset quality.

Link To Code: https://github.com/zihengh1/Grad-Mimic

Primary Area: General Machine Learning->Everything Else

Keywords: Weight-Space Geometry, Data Selection, Data Curation, Data Quality Metric

Submission Number: 9039