Data Ambiguity Strikes Back: How Documentation Improves GPT's Text-to-SQL

Published: 28 Oct 2023, Last Modified: 23 Nov 2023TRL @ NeurIPS 2023 OralEveryoneRevisionsBibTeX
Keywords: Text-to-SQL, data quality, data ambiguities, GPT, documentation, data management
TL;DR: We study data ambiguities prevalent in real-world datasets for Text-to-SQL but are often overlooked. By providing documentation on value consistency, data coverage, and granularity, GPT-4's performance in Text-to-SQL tasks improved by 28.9%.
Abstract: Text-to-SQL allows experts to use databases without in-depth knowledge of them. However, real-world tasks have both query and data ambiguities. Most works on Text-to-SQL focused on query ambiguities and designed chat interfaces for experts to provide clarifications. In contrast, the data management community has long studied data ambiguities, but mainly addresses error detection and correction, rather than documenting them for disambiguation in data tasks. This work delves into these data ambiguities in real-world datasets. We have identified prevalent data ambiguities of value consistency, data coverage, and data granularity that affect tasks. We examine how documentation, originally made to help humans to disambiguate data, can help GPT-4 with Text-to-SQL tasks. By offering documentation on these, we found GPT-4's performance improved by $28.9$%.
Slides: pdf
Submission Number: 20