ToolDec: Syntax Error-Free and Generalizable Tool Use for LLMs via Finite-State Decoding

Kexun Zhang, Hongqiao Chen, Lei Li, William Yang Wang

Primary Area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: large language models, augmented language models, finite-state machines

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We propose ToolDec, a decoding algorithm guided by a finite-state machine to make LLM tool use without syntax errors and generalizable.

Abstract:

Large language models (LLMs) have shown promising capabilities in using external tools to solve complex problems.
However, existing approaches either involve fine-tuning on tool demonstrations, which does not generalize to new tools without additional training, or providing tool documentation in context, limiting the number of tools. Both approaches often generate syntactically invalid tool calls.
In this paper, we propose ToolDec, a finite-state machine-guided decoding algorithm for tool-augmented LLMs.
ToolDec eliminates tool-related errors for any tool-augmented LLMs by ensuring valid tool names and type-conforming arguments.
Furthermore, ToolDec enables LLM to effectively select tools using only the information contained in their names, with no need for fine-tuning or in-context documentation.
We evaluated multiple prior methods and their ToolDec-enhanced versions on a variety of tasks involving tools like math functions, knowledge graph relations, and complex real-world RESTful APIs.
Our experiments show that ToolDec reduces syntactic errors to zero, consequently achieving significantly better performance and as much as a 2x speedup.
We also show that ToolDec achieves superior generalization performance on unseen tools, performing up to 8x better than the baselines

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6402