Tree-guided Token Decoding for SQL Generation: Augmenting LLM Decoding for Jointly Reduced Latency and Hallucination

ACL ARR 2024 December Submission2052 Authors

Abstract:

Grammar constrained decoding (GCD) empowers LLMs to generate highly structured outputs such as programming code (ex: SQL, Python, Go, etc.). While GCD is effective in generating structurally valid code \cite{suynchromesh-iclr-2022,syncode-arxiv-2024} and it does not require fine-tuning, this line of work has two notable limitations: (a) these methods demand high inference times as compared to that of the unconstrained auto-regressive decoding setting, making them unfit for real-time code generation applications (ex: AI Assistants and Co-pilots); and (b) the final accuracy of code generation tasks are low as compared to the fraction of structurally valid generated code.

In this paper, we tackle the above research gap particularly in the context of SQL generation task. By observing that
the accuracy of standard auto-regressive LLM decoding methods in generating structurally valid SQL code is on par with that of GCD methods, we propose a novel unified approach $-$ we refer to as {\em Tree-guided Token Decoding (TTD)} $-$ which guides LLMs in decoding SQL-keywords and database schema items (i.e. names of tables and columns) without paying attention to the SQL grammar. Guiding schema items reduces hallucination errors and such guiding often results in auto-filling certain tokens without explicit LLM calls, resulting in reduced inference times. We conduct thorough experiments using two popular datasets (Spider and BIRD) to demonstrate the efficacy of our proposed approach using two metrics: execution accuracy and token rate.

Paper Type: Long

Research Area: Machine Learning for NLP

Research Area Keywords: SQL Generation, Text to SQL, LLM decoding, Speculative Decoding

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Approaches to low-resource settings

Languages Studied: English, SQL

Submission Number: 2052