Data-Efficient LLM Fine-Tuning by Noise Resistance

ACL ARR 2025 May Submission7216 Authors

Abstract: Fine-tuning large language models (LLMs) remains challenging due to the scarcity of downstream task data and the prevalence of noisy supervision. In this paper, we introduce Noise Flushing (NF), a novel paradigm that prioritizes noise elimination over data augmentation. NF leverages abundant irrelevant data—sampled from the base LLM—to mitigate noise and sharpen focus on task-relevant signals during fine-tuning, thus enabling effective adaptation in extremely low-resource settings. Theoretically, we show that NF can match or even surpass the performance of standard LoRA finetuning settings, despite using substantially fewer task-specific examples. Empirically, NF achieves consistent and significant improvements over strong fine-tuning baselines across various tasks, including machine translation, structured text generation, text-to-SQL, and special token understanding—even with fewer than 100 examples.

Paper Type: Long

Research Area: Efficient/Low-Resource Methods for NLP

Research Area Keywords: data-efficient training, NLP in resource-constrained settings

Contribution Types: Approaches to low-resource settings

Languages Studied: english

Submission Number: 7216