ZIP-FIT: Embedding-Free Data Selection via Compression-Based Alignment
Keywords: data centric machine learning, autoformalization, large language models, reasoning
TL;DR: use gzip to select optimal data for code and autoformalization
Abstract: Data selection is crucial for optimizing language model (LM) performance on
specific tasks, yet most existing methods fail to effectively consider the target task
distribution. Current approaches either ignore task-specific requirements entirely
or rely on approximations that fail to capture the nuanced patterns needed for tasks
like Autoformalization or code generation. Methods that do consider the target
distribution often rely on simplistic, sometimes noisy, representations, like hashed
n-gram features, which can lead to collisions and introduce noise. We introduce
ZIP-FIT, a data selection framework that uses gzip compression to directly
measure alignment between potential training data and the target task distribution. Our key insight is that compression-based similarity captures both syntactic
and structural patterns relevant to the target task, enabling more precise selection of truly task-relevant data. In extensive evaluations on Autoformalization and
Python code generation, ZIP-FIT significantly outperforms leading baselines
like DSIR and D4. Models trained on ZIP-FIT-selected data achieve their lowest cross-entropy loss up to 85.1% faster than baselines, demonstrating that better
task alignment leads to more efficient learning. In addition, ZIP-FIT performs
selection up to 65.8% faster than DSIR and two orders of magnitude faster than
D4. Notably, ZIP-FIT shows that smaller, well-aligned datasets often outperform larger but less targeted ones, demonstrating that a small amount of higher
quality data is superior to a large amount of lower quality data. Our results imply that task-aware data selection is crucial for efficient domain adaptation, and
that compression offers a principled way to measure task alignment. By showing
that targeted data selection can dramatically improve task-specific performance,
our work provides new insights into the relationship between data quality, task
alignment, and model learning efficiency.
Primary Area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 8917
Loading