ToolDec: Syntax Error-Free and Generalizable Tool Use for LLMs via Finite-State Decoding

Hongqiao Chen, Kexun Zhang, Lei Li, William Yang Wang

Keywords: large language models, augmented language models, finite-state machines

Abstract:

Large language models (LLMs) have shown promising capabilities in using external tools.
However, existing approaches rely on fine-tuning or in-context learning to use tools, which make syntactic mistakes and are difficult to generalize.
In this paper, we propose ToolDec, a finite-state machine-guided decoding algorithm for tool-augmented LLMs.
ToolDec eliminates tool-related errors by ensuring valid tool names and type-conforming arguments.
Furthermore, ToolDec enables LLM to effectively select tools using only the information contained in their names, with no need for tool-specific fine-tuning.
Our experiments on multiple word problem datasets show that ToolDec reduces syntactic errors to zero, consequently achieving significantly better performance and as much as a 2x speedup.
We also show that ToolDec achieves superior generalization performance on unseen tools, performing up to 8x better than the baseline.

Submission Number: 49