Influence-driven Curriculum Learning for Pre-training on Limited Data

ACL ARR 2025 May Submission3311 Authors

Abstract:

Curriculum learning, a training technique where data is presented to the model in order of example difficulty (e.g., from simpler to more complex documents), has shown limited success for pre-training language models.
In this work, we investigate whether curriculum learning becomes competitive if we replace conventional human-centered difficulty metrics with one that more closely corresponds to example difficulty as observed during model training.
Specifically, we experiment with sorting training examples by their *training data influence*, a score which estimates the effect of individual training examples on the model's output.
Models trained on our curricula are able to outperform ones trained in random order by over 10 percentage points in benchmarks, confirming that curriculum learning is beneficial for language model pre-training, as long as a more model-centric notion of difficulty is adopted.

Paper Type: Long

Research Area: Efficient/Low-Resource Methods for NLP

Research Area Keywords: data-efficient training,LLM Efficiency,NLP in resource-constrained settings

Contribution Types: Model analysis & interpretability, Approaches to low-resource settings, Publicly available software and/or pre-trained models

Languages Studied: eng

Submission Number: 3311