Data-Efficient LLM Fine-Tuning by Noise Resistance

ACL ARR 2025 February Submission8479 Authors

Abstract: Fine-tuning large language models (LLMs) with limited data is challenging due to the entanglement of task features and sample-specific noise. We introduce Noise Flushing, a paradigm shift that prioritizes removing noise rather than solely amplifying the weak task signal. Like gold panning, where water washes away sand, Noise Flushing uses abundant irrelevant data – sampled from the LLM itself – to "flush away" noise during LoRA fine-tuning. This constrains the LoRA adapter to suppress noise and focus on task-relevant features. Theoretically, we show that Noise Flushing can achieve performance comparable to vanilla fine-tuning with drastically fewer task samples. Empirically, with remarkably few task examples, Noise Flushing achieves significant improvements over strong fine-tuning baselines on translation, structured text generation, even special token understanding with fewer than 100 samples. Noise Flushing transforms LLMs into statistical "gold panners", learning what to ignore to efficiently learn from sparse data.

Paper Type: Long

Research Area: Efficient/Low-Resource Methods for NLP

Research Area Keywords: data-efficient training, NLP in resource-constrained settings

Contribution Types: Approaches to low-resource settings

Languages Studied: English

Submission Number: 8479