NERQual: Evaluating the Robustness of Named Entity Recognition Models to Data Quality Issues

ACL ARR 2026 January Submission2092 Authors

Keywords: Named Entity Recognition, NER robustness, data quality, noisy text, transformer models, adversarial perturbations, robustness evaluation, orthographic noise, syntactic noise, semantic perturbations, noise-aware training

Abstract: Transformer-based models for Named Entity Recognition (NER) achieve strong performance on clean benchmarks, yet their reliability in real-world settings remains insufficiently understood. In practical applications, NER systems are frequently exposed to degraded input originating from optical character recognition, automatic speech recognition, and user-generated text. We address this gap by deriving a taxonomy of data quality perturbations from a systematic literature review, consolidating fragmented prior work into a unified framework. Guided by this taxonomy, we conduct controlled experiments evaluating six widely used transformer architectures under varying perturbation types, intensities, and training configurations. Our results show that syntactic perturbations cause the most severe performance degradation across models, while architectural features such as character-level processing and disentangled attention confer specific robustness advantages. Furthermore, we demonstrate that targeted training with perturbed data can recover more than half of the lost performance with minimal impact on clean-data accuracy.

Paper Type: Long

Research Area: Resources and Evaluation

Research Area Keywords: benchmarking,evaluation methodologies, reproducibility data influence, adversarial examples,robustness, named entity recognition

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Data analysis

Languages Studied: English

Submission Number: 2092