Coarse and Fine-grained Confidence Calibration of LLM-based Text-to-SQL Generation
Abstract: Calibration plays a crucial role as LLMs are increasingly deployed to convert natural language questions into SQL over commercial databases. In this work, we study the calibration of the confidence attached to both the whole query, and for the first time, to sub-parts of the query. For whole queries, we demonstrate that the simple baseline of deriving confidence from model assigned whole sequence probability yields the best calibration surpassing recent self-check and verbalization methods. For fine-grained calibration, we propose a novel method of assigning confidence to nodes of a logical relational algebra tree representation of the SQL string. We present an extensive comparison spanning two popular Text-to-SQL benchmarks on multiple LLMs, and draw interesting insights about various calibration methods.
Paper Type: Long
Research Area: Interpretability and Analysis of Models for NLP
Research Area Keywords: calibration/uncertainty, code generation and understanding, language model, Relational Algebra Tree, SQL
Contribution Types: Model analysis & interpretability, NLP engineering experiment
Languages Studied: English
Submission Number: 5862
Loading