Exploring Chain of Thought Style Prompting for Text-to-SQL
Submission Type: Regular Long Paper
Submission Track: Semantics: Lexical, Sentence level, Document Level, Textual Inference, etc.
Submission Track 2: Question Answering
Keywords: Text-to-Sql, Prompt engineering, Chain-of-Thought, Least-to-Most, Question Decomposition.
TL;DR: We systematically study how to enhance the reasoning ability of large language models for text-to-SQL parsing through chain of thought style prompting.
Abstract: In-context learning with large language models (LLMs) has recently caught increasing attention due to its superior few-shot performance on various tasks. However, its performance on text-to-SQL parsing still has much room for improvement. In this paper, we hypothesize that a crucial aspect of LLMs to improve for text-to-SQL parsing is their multi-step reasoning ability. Thus, we systematically study how to enhance LLMs' reasoning ability through chain of thought (CoT) style prompting, including the original chain-of-thought prompting and least-to-most prompting. Our experiments demonstrate that iterative prompting as in least-to-most prompting may be unnecessary for text-to-SQL parsing, and using detailed reasoning steps tends to have more error propagation issues. Based on these findings, we propose a new CoT-style prompting method for text-to-SQL parsing. It brings 5.2 and 6.5 point absolute gains on the Spider development set and the Spider Realistic set, respectively, compared to the standard prompting method without reasoning steps; 2.4 and 1.5 point absolute gains, compared to the least-to-most prompting method.
Submission Number: 4741
Loading