Reducing Tool Hallucination via Reliability Alignment

Hongshen Xu, Zichen Zhu, Lei Pan, Zihan Wang, Su Zhu, Da Ma, Ruisheng Cao, Lu Chen, Kai Yu

Abstract: Large Language Models (LLMs) have expanded their capabilities beyond language generation to interact with external tools, enabling automation and real-world applications. However, tool hallucinations—where models either select inappropriate tools or misuse them—pose significant challenges, leading to erroneous task execution, increased computational costs, and reduced system reliability. To systematically address this issue, we define and categorize tool hallucinations into two main types: tool selection hallucination and tool usage hallucination. To evaluate and mitigate these issues, we introduce RelyToolBench, which integrates specialized test cases and novel metrics to assess hallucination-aware task success and efficiency. Finally, we propose Relign, a reliability alignment framework that expands the tool-use action space to include indecisive actions, allowing LLMs to defer tool use, seek clarification, or adjust tool selection dynamically. Through extensive experiments, we demonstrate that Relign significantly reduces tool hallucinations, improves task reliability, and enhances the efficiency of LLM tool interactions. The code and data will be publicly available.

Lay Summary: Large language models (LLMs), like the ones behind AI chatbots, are becoming smarter — they can now use external tools such as calculators or search engines to solve real-world tasks. But sometimes, they make mistakes when using these tools. For example, they might pick the wrong tool or use it in the wrong way, leading to wrong answers, wasted resources, or unreliable systems. To better understand and fix this issue, we first define two types of tool mistakes: picking the wrong tool (selection errors) and using the right tool incorrectly (usage errors). We then introduce a new evaluation set called RelyToolBench that tests how well models use tools and how often they make these mistakes. Building on this, we propose a new method called Relign, which teaches models to be more cautious. Instead of rushing to use a tool, the model can now say “I’m not sure,” ask for more information, or try a different tool. Our experiments show that this approach makes tool use more accurate, reliable, and efficient, helping AI systems become more trustworthy in practical applications.

Application-Driven Machine Learning: This submission is on Application-Driven Machine Learning.

Link To Code: https://github.com/X-LANCE/ToolHallucination

Primary Area: Applications->Language, Speech and Dialog

Keywords: Tool hallucination; Reliability Alignment; Large Language Model

Submission Number: 10507