Multi-agent KTO: Enhancing Strategic Interactions of Large Language Model in Language Game

Rong Ye, Yongxin Zhang, Yikai Zhang, Haoyu Kuang, peng sun, zhongyu wei

Keywords: Large Language Model, Language Game, Application

TL;DR: We introduced Multi-agent KTO, a method that trains LLM to play Werewolf through direct gameplay. Our approach outperforms GPT-4o and RL+LLM methods, achieving human-competitive performance.

Abstract: Achieving Artificial General Intelligence (AGI) requires AI agents that can not only make strategic decisions but also engage in flexible and meaningful communication. Inspired by Wittgenstein's language game theory, we propose that language agents can learn through in-context interaction rather than traditional multi-stage frameworks that separate decision-making from language expression. Using Werewolf, a social deduction game that tests language understanding, strategic interaction, and adaptability, as a test bed, we develop the Multi-agent Kahneman-Tversky's Optimization (MaKTO). MaKTO engages diverse models in extensive gameplay to generate unpaired desirable and unacceptable responses, then employs KTO to refine the model's decision-making process. In 9-player Werewolf games, MaKTO achieves a 61% average win rate across various models, outperforming GPT-4o and two-stage RL agents by relative improvements of 23.0% and 10.9%, respectively. Notably, MaKTO also demonstrates human-like performance, winning 60% against expert players and showing only 48.9% detectability in Turing-style blind tests. Code and data are available at project page https://reneeye.github.io/MaKTO.html.

Primary Area: Applications (e.g., vision, language, speech and audio, Creative AI)

Submission Number: 11638