Token Hijacking Improves In-Context Tool Use

Jie S. Li, David Yu Miller, Ashwinee Panda, Tom Goldstein

Keywords: LLM tool use

TL;DR: more efficient and effective tool use

Abstract:

Enabling LLMs to interact with external tools–such as APIs, databases, or computational services–remains a significant challenge. Pre-trained LLMs often fail to call external tools in zero-shot scenarios, requiring augmented vocabulary and fine-tuning on sometimes-sensitive data.
We offer a method that addresses these concerns with only in-context learning of tool use, using existing vocabulary tokens. This method enables dynamic and extensible integration of tools without additional fine-tuning. It also avoids the avoids the significant overhead and privacy concerns that can arise with fine-tuning. We introduce the use of specialized trigger tokens–referred to as metatokens–to reliably elicit tool-using behavior. We describe a procedure for identifying effective metatokens for a given tool, and we empirically demonstrate that this technique significantly improves tool-use performance.

Submission Number: 29