Drill-Down Analysis of LLM Hallucination Patterns in Text-to-SQL

ICLR 2026 Conference Submission23522 Authors

Keywords: Drill-Down, Hallucination, Decomposition, Text-to-SQL, Error Propagation

TL;DR: This paper proposes a drill-down evaluation framework to analyze hallucination patterns of large language models (LLMs) in Text-to-SQL by decomposing complex queries into progressive sub-queries and providing key insights into their failure modes.

Abstract: Despite impressive benchmark scores, Large Language Models (LLMs) can still produce flawed and incorrect responses for Text-to-SQL tasks. While prior work has decomposed complex SQL queries in an attempt to improve LLM benchmark performance, few have systematically analyzed hallucination propagation patterns within these decomposed structures. We present a drill-down evaluation framework that decomposes complex SQL queries and questions from the BIRD-mini dataset Li et al. (2023), allowing for a fine-grained analysis of hallucination propagation. Through our analysis, we report three key findings: (1) Recurrent Hallucinations: Many hallucinations persistently propagate from early, structurally simple sub-queries through to final steps, indicating systematic misalignment. (2) Final-Step Emergence: Fewer, but specific hallucination types emerge in the final step, suggesting a distinct failure mode tied to query complexity. (3) History Amplifies Recurrence: While contextual information between sub-queries can help to reduce the frequency of emergent hallucinations, it consequently increases the recurrence of early-stage hallucinations. This framework establishes a methodology to better understand LLM weaknesses and failure modes for Text-to-SQL systems.

Primary Area: interpretability and explainable AI

Submission Number: 23522