The Large Language Model aided expert problem

ACL ARR 2025 February Submission2228 Authors

Abstract: Large Language Models (LLMs) often falter in providing accurate responses to queries that demand up-to-date or context-specific information. Retrieval-Augmented Generation (RAG) addresses this limitation by incorporating a retriever to fetch relevant documents from databases or the Internet. However, RAG falls short when relevant information is unavailable, necessitating expert intervention—a process that is both costly and inefficient. This work introduces and addresses the {\it LLM-aided expert problem}, aiming to develop systems that progressively enhance their competence in answering queries while minimizing the need for expert input. We propose two decision-making strategies: (1) a classifier-based approach that employs threshold-based filtering to evaluate retrieved answers, and (2) a contextual bandit approach that models the decision to rely on retrieved answers or escalate to an expert as a two-arm bandit problem. Both methods utilize Pretrained Language Models for answer validation and reward estimation. We evaluate these strategies using a benchmark derived from the Quora Question Pairs dataset, demonstrating their effectiveness in reducing expert interventions while maintaining high accuracy. Our results highlight the potential of adaptive decision-making frameworks to enhance LLM reliability in dynamic query-answering environments.

Paper Type: Long

Research Area: Question Answering

Research Area Keywords: Question Answering

Contribution Types: NLP engineering experiment, Data resources

Languages Studied: English

Submission Number: 2228