Out of Style: RAG’s Fragility to Linguistic Variation

ACL ARR 2025 July Submission501 Authors

Abstract:

Despite the impressive performance of Retrieval-augmented Generation (RAG) systems across various NLP benchmarks, their robustness in handling real-world user-LLM interaction queries remains largely underexplored.
This presents a critical gap for practical deployment, where user queries exhibit greater linguistic variations and can trigger cascading errors across interdependent RAG components.
In this work, we systematically analyze how varying four linguistic dimensions
(formality, readability, politeness, and grammatical correctness)
impact RAG performance.
We evaluate two retrieval models and nine LLMs, ranging from 3 to 72 billion parameters, across four information-seeking Question Answering (QA) datasets.
Our results reveal that linguistic reformulations significantly impact both retrieval and generation stages, leading to a relative performance drop of up to 40.41\% in Recall@5 scores for less formal queries and 38.86\% in answer match scores for queries containing grammatical errors.
Notably, RAG systems exhibit greater sensitivity to such variations compared to LLM-only generations, highlighting their vulnerability to error propagation due to linguistic shifts.
These findings highlight the need for improved robustness techniques to enhance reliability in diverse user interactions.

Paper Type: Long

Research Area: Resources and Evaluation

Research Area Keywords: evaluation;automatic creation and evaluation of language resources

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English

Reassignment Request Area Chair: This is not a resubmission

Reassignment Request Reviewers: This is not a resubmission

A1 Limitations Section: This paper has a limitations section.

A2 Potential Risks: N/A

B Use Or Create Scientific Artifacts: Yes

B1 Cite Creators Of Artifacts: Yes

B1 Elaboration: 4

B2 Discuss The License For Artifacts: Yes

B2 Elaboration: Appendix D

B3 Artifact Use Consistent With Intended Use: Yes

B3 Elaboration: Appendix D

B4 Data Contains Personally Identifying Info Or Offensive Content: N/A

B5 Documentation Of Artifacts: N/A

B5 Elaboration: 3, 4

B6 Statistics For Data: Yes

B6 Elaboration: 3, 4

C Computational Experiments: Yes

C1 Model Size And Budget: Yes

C1 Elaboration: Appendix L

C2 Experimental Setup And Hyperparameters: Yes

C2 Elaboration: 4, Appendix B.1

C3 Descriptive Statistics: Yes

C3 Elaboration: 5

C4 Parameters For Packages: Yes

C4 Elaboration: 4, Appendix A.1

D Human Subjects Including Annotators: No

D1 Instructions Given To Participants: N/A

D2 Recruitment And Payment: N/A

D3 Data Consent: N/A

D3 Elaboration: N/A

D4 Ethics Review Board Approval: N/A

D5 Characteristics Of Annotators: N/A

E Ai Assistants In Research Or Writing: Yes

E1 Information About Use Of Ai Assistants: Yes

E1 Elaboration: I used ChatGPT to help polish the paper writing.

Author Submission Checklist: yes

Submission Number: 501