LLMs in the Real World: Evaluating “AI” in Emergency Contexts

ACL ARR 2026 January Submission436 Authors

Keywords: Ethics, Bias, and Fairness, Ethical considerations in NLP applications, Human-Centered NLP, Low-resource NLP, Machine Translation, MT deployment and maintenance, NLP Applications, NLP for social good, Policy and governance, Reflections and critiques, Transparency

Abstract: We present a case study on the initial stages of an LLM-based machine translation system's deployment in a real-world context: a text-2-911 system advertising capabilities in 55 languages for use in emergencies in which it may be difficult to call operators directly. We identify a number of common misconceptions about these technologies and describe their implications, concluding with a set of concrete recommendations and best practices for stakeholders at every stage of the development and deployment pipeline. We offer a call to action and urge our colleagues in the research community to play a greater role in the articulation of our findings to the public. While the advancement of scientific research often lies in solving the "hard" problems, we argue that it is often the "easy" ones -- problems for which the latest technology is often unnecessary -- that are most overlooked.

Paper Type: Long

Research Area: Ethics, Bias, and Fairness

Research Area Keywords: Ethics, Bias, and Fairness, Ethical considerations in NLP applications, Human-Centered NLP, Machine Translation, MT deployment and maintenance, NLP Applications, NLP for social good, Policy and governance, Reflections and critiques, Transparency

Contribution Types: Approaches to low-resource settings, Position papers

Languages Studied: N/A

Submission Number: 436