CPO-SQL: Boosting Small LLMs for Text-to-SQL via Efficient In-Context Learning and Preference Optimization
Abstract: Most recent researches in Text-to-SQL parsing overly rely on the proprietary Large Language Models (LLMs), raising concerns of data privacy and inference overheads. To narrow the gap between small LLMs and proprietary LLMs in Text-to-SQL, we introduce CPO-SQL, an approach aiming to efficiently boost the capability of small LLMs via In-Context Learning and Preference Optimization. This approach builds the enhanced training set by sampling demonstrations from beta distribution based on the similarity of questions and SQL, and then fine-tune the small LLMs to empower them with ICL capabilities of Text-to-SQL. Further, we propose a new Spider preference set, constructed by an agile semi-automated process, based on six types of SQL optimization. On this basis, we employ SFT-enhanced preference optimization to support the mixed training on the supervised set and the preference set, enabling us to optimize the SQL generation in complex query scenarios while maintaining the learning of original data. By this way, we can balance the generation ability of small LLMs for questions of varying difficulty. Finally, we evaluate our method on Spider and its three robustness-diagnostic variants, shedding light on the strengths and weaknesses of it.
Paper Type: Long
Research Area: NLP Applications
Research Area Keywords: NLP Applications, Generation
Contribution Types: NLP engineering experiment, Approaches to low-resource settings, Approaches low compute settings-efficiency, Data resources
Languages Studied: English, SQL
Submission Number: 1429
Loading