Evaluating Ambiguous Questions in Text2SQL
Keywords: Semantic Parsing, Text2SQL, Ambiguity, Automatic Generation
TL;DR: We introduce a Data-Ambiguity Tester that injects realistic schema ambiguities into clear Text2SQL queries, revealing where models struggle with ambiguous natural language.
Abstract: Recent advancements in Tabular Representation Learning (TRL) and Large Language Models (LLMs) have achieved promising results in the Text2SQL task, which involves converting natural language questions about relational tables into executable SQL queries. However, when questions are ambiguously defined to the table schema, existing models often fail to produce correct outputs. Assessing the robustness of such data ambiguity is labor-intensive, as it requires identifying ambiguous patterns across many queries with varying levels of complexity. To address this challenge, we introduce the Data-Ambiguity Tester, a dedicated pipeline designed for ambiguous Text2SQL generation. This approach first generates a diverse set of unambiguous questions alongside their corresponding SQL queries. It then methodically injects ambiguous patterns from a human-annotated set of relational tables into these questions, simulating realistic schema ambiguities. Finally, the pipeline employs customized metrics to evaluate Text2SQL model performance under ambiguity. Our experimental results provide valuable insights into the strengths and limitations of current Text2SQL models.
Submission Number: 17
Loading