iTool: Boosting Tool Use of Large Language Models via Iterative Reinforced Fine-Tuning

ACL ARR 2025 February Submission302 Authors

Abstract:

Augmenting large language models (LLMs) with external tools is known as a promising approach to enhancing their capabilities, especially for complex tasks.
It hinges crucially on improving their ability to use tools.
Synthesizing tool-use data through real-world simulations is an effective way to enhance this ability.
Nevertheless, our investigation reveals that (1) training gains significantly decay as synthetic data increases.
The model struggles to benefit from more synthetic data due to potential data diversity issues, resulting in poor performance in complex scenarios.
Moreover, we find that (2)
this challenge primarily manifests as minor discrepancies between the model's output and the ground truth response (termed as deficiency), such as errors in parameter values that require complex reasoning from the context to resolve.
To this end, we propose an iterative reinforced fine-tuning strategy designed to alleviate these challenges.
This strategy involves: (1) enhancing the diversity of synthetic data through path exploration of Monte Carlo Tree Search.
(2) iteratively identifying deficiency-related data, constructing fine-grained preference pairs to pinpoint deficiencies, and then applying preference optimization to optimize these deficiencies.
Our experiments show that models trained using our method achieve about 3\% better performance than same-size models, outperforming larger open-source and closed-source models.

Paper Type: Long

Research Area: NLP Applications

Research Area Keywords: LLM, tool use, preference learning

Contribution Types: NLP engineering experiment

Languages Studied: English

Submission Number: 302