Token Hijacking Improves In-Context Tool Use

ACL ARR 2025 May Submission1490 Authors

Abstract:

While large language models (LLMs) have
demonstrated impressive capabilities across a
range of natural language tasks, enabling them
to interact with external tools–such as APIs,
databases, or computational services–remains a
significant challenge. Pre-trained LLMs often
perform poorly in zero-shot tool-use scenarios,
lacking the structure or inductive bias neces-
sary to reliably call external tools. Fine-tuning
models on tool-use datasets can yield strong
performance, but such models are inherently
limited to the tools included during training,
and extending them to new tools requires costly
retraining. This approach is also problematic
in domains involving private or sensitive tool-
related data, where fine-tuning may raise pri-
vacy or security concerns. Therefore, there
is a critical need for methods that enable ef-
fective, extensible, and privacy-preserving tool
use without requiring additional training or fine-
tuning.
We offer a method that addresses these con-
cerns with in-context learning of tool use using
metatokens. This method enables dynamic and
extensible integration of tools without requiring
additional model fine-tuning. This approach
supports greater customizability, allowing new
tools to be added simply by updating the input
context, rather than retraining the model. It
is also computationally efficient, avoiding the
significant overhead and privacy concerns as-
sociated with fine-tuning, especially in scenar-
ios involving proprietary or sensitive data. We
introduce the use of specialized trigger tokens–
referred to as metatokens–to reliably elicit tool-
using behavior from the model. We describe a
procedure for identifying effective metatokens
for a given tool, and we empirically demon-
strate that this technique significantly improves
tool-use performance.

Paper Type: Long

Research Area: Language Modeling

Research Area Keywords: AI Agents

Contribution Types: NLP engineering experiment

Languages Studied: English

Submission Number: 1490