Finetuning LLMs for Text-to-SQL with Two-Stage Progressive Learning

Published: 01 Jan 2024, Last Modified: 17 May 2025NLPCC (2) 2024EveryoneRevisionsBibTeXCC BY-SA 4.0
Abstract: With the widespread usage of large language model (LLMs), LLM-based method has become the mainstream approach for Text-to-SQL tasks, achieving leading performance on Text-to-SQL leaderboards. However, generating complex SQL queries correctly has always been a main challenge. Current LLM-based models primarily utilize prompting-based methods on large scale closed-source LLMs (e.g., GPT-4 and ChatGPT), which may cause concerns of usage costs and data privacy. For fine-tuning based methods, it is difficult to generate complex SQL accurately in only one fine-tuning step. Focusing on this, we propose TSP-SQL, a Two-Stage Progressive learning method for Text-to-SQL. TSP-SQL decomposes Text-to-SQL task into two stages: SQL elements generation auxiliary task, and SQL query generation main task. The two tasks are progressively fine-tuned on a single model, effectively reducing the difficulty of SQL generation and improving accuracy. TSP-SQL achieves state-of-the-art performance among open-source fine-tuning based methods on Spider dev set, and surpasses most of the methods based on large scale closed-source LLMs.
Loading