COLT: Enhancing Video Large Language Models with Continual Tool Usage

TMLR Paper6392 Authors

Abstract: The success of Large Language Models (LLMs) has significantly propelled the research of video understanding. To harvest the benefits of well-trained expert models (i.e., tool), video LLMs prioritize the exploration of tool usage capabilities. Existing methods either prompt closed-source LLMs or employ the instruction tuning paradigm for tool-use finetuning. These methods, however, assume an established repository of fixed tools and struggle to generalize to real-world environments where tool data is perpetually evolving and streaming in. To this end, we propose to enhance open-source video LLMs with COntinuaL Tool usage (termed COLT), which automatically acquires tool-use ability in a successive tool stream without suffering "catastrophic forgetting" of the past learned tools. Specifically, our COLT incorporates a learnable tool codebook as a tool-specific memory system. Then, relevant tools are dynamically selected based on the similarity between user instructions and tool features within the codebook. To unleash the tool usage potential of video LLMs, we collect a video-centric tool-use instruction tuning dataset VideoToolBench. Extensive experiments on both previous video LLM benchmarks and the tool-use-specific VideoToolBench dataset demonstrate the state-of-the-art performance of our proposed COLT.

Submission Type: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission: The previous submission (https://openreview.net/pdf?id=Qes0Bzry6W) was desk-rejected due to formatting issues. We have corrected the font and ensured full compliance with the official TMLR template. We do not provide the direct OpenReview forum link here as the original submission page is not anonymous. (I can see my name when I click into it)

Assigned Action Editor: ~Yuhang_Zang1

Submission Number: 6392