Measuring the Robustness of NLP Models to Domain Shifts
Abstract: Existing research on Domain Robustness (DR) suffers from disparate setups, limited task variety, and scarce research on recent capabilities such as in-context learning. Furthermore, the common practice of measuring DR might not be fully accurate. Current research focuses on challenge sets and relies solely on the Source Drop (SD): Using the source in-domain performance as a reference point for degradation.
However, we argue that the Target Drop (TD), which measures degradation from the target in-domain performance, should be used as a complementary point of view.
To address these issues, we first curated a DR benchmark comprised of 7 diverse NLP tasks, which enabled us to measure both the SD and the TD.
We then conducted a comprehensive large-scale DR study involving over 14,000 domain shifts across 21 fine-tuned models and few-shot LLMs.
We found that both model types suffer from drops upon domain shifts. While fine-tuned models excel in-domain, few-shot LLMs often surpass them cross-domain, showing better robustness.
In addition, we found that a large SD can often be explained by shifting to a harder domain rather than by a genuine DR challenge, and this highlights the importance of TD as a complementary metric.
We hope our study will shed light on the current DR state of NLP models and promote improved evaluation practices toward more robust models.
Paper Type: Long
Research Area: Interpretability and Analysis of Models for NLP
Research Area Keywords: robustness, domain adaptation
Contribution Types: Model analysis & interpretability, NLP engineering experiment, Data resources, Data analysis, Position papers, Theory
Languages Studied: English
Submission Number: 700
Loading