Is This Collection Worth My LLM's Time? Automatically Measuring Information Potential in Text Corpora

ACL ARR 2025 February Submission2222 Authors

Abstract: As large language models (LLMs) converge towards similar capabilities, the key to advancing their performance lies in identifying and incorporating valuable new information sources. However, evaluating which text collections are worth the substantial investment required for digitization, preprocessing, and integration into LLM systems remains a significant challenge. We present a novel approach to this challenge: an automated pipeline that evaluates the potential information gain from text collections without requiring model training or fine-tuning. Our method generates multiple choice questions (MCQs) from texts and measures an LLM's performance both with and without access to the source material. The performance gap between these conditions serves as a proxy for the collection's information potential. We validate our approach using three strategically selected datasets: EPFL PhD manuscripts (likely containing novel specialized knowledge), Wikipedia articles (presumably part of training data), and a synthetic baseline dataset. Our results demonstrate that this method effectively identifies collections containing valuable novel information, providing a practical tool for prioritizing data acquisition and integration efforts.

Paper Type: Long

Research Area: Resources and Evaluation

Research Area Keywords: LLM evaluation, information gain assessment, multiple choice question generation, performance gap analysis, dataset evaluation, novel knowledge detection, automated evaluation pipeline

Contribution Types: Data analysis

Languages Studied: english

Submission Number: 2222