Influence Scores at Scale for Efficient Language Data Sampling

Nikhil Anand, Joshua Tan, Maria Minakova

Submission Type: Regular Long Paper

Submission Track: Efficient Methods for NLP

Submission Track 2: Machine Learning for NLP

Keywords: data effiency, data sampling, difficulty metrics, influence scores, pruning

TL;DR: We can train more data-efficient language models by intelligently choosing a subset of training data. We implement this procedure in public models/datasets as well as in an industry-scale setting and document the results/challenges of doing so.

Abstract:

Modern ML systems ingest data aggregated from diverse sources, such as synthetic, human-annotated, and live customer traffic. Understanding \textit{which} examples are important to the performance of a learning algorithm is crucial for efficient model training. Recently, a growing body of literature has given rise to various “influence scores,” which use training artifacts such as model confidence or checkpointed gradients to identify important subsets of data. However, these methods have primarily been developed in computer vision settings, and it remains unclear how well they generalize to language-based tasks using pretrained models.

In this paper, we explore the applicability of influence scores in language classification tasks. We evaluate a diverse subset of these scores on the SNLI dataset by quantifying accuracy changes in response to pruning training data through random and influence-score-based sampling. We then stress-test one of the scores – "variance of gradients" (VoG) from Agarwal and Hooker (2022) – in an NLU model stack that was exposed to dynamic user speech patterns in a voice assistant type of setting. Our experiments demonstrate that in many cases, encoder-based language models can be fine-tuned on roughly 50% of the original data without degradation in performance metrics. Along the way, we summarize lessons learned from applying out-of-the-box implementations of influence scores, quantify the effects of noisy and class-imbalanced data, and offer recommendations on score-based sampling for better accuracy and training efficiency.

Submission Number: 3680