Deviation Ratings: A general, clone invariant rating method

Luke Marris, Siqi Liu, Ian Gemp, Georgios Piliouras, Marc Lanctot

Keywords: rating, ranking, coarse correlated equilibria, Nash equilibria, game theory, equilibria, LLM leaderboard, normal-form game

TL;DR: An n-player general-sum clone-invariant rating scheme for rating strategies in strategic interactions. Many real-world scenarios can be rated in such a scheme including LLMs, agents, and tasks.

Abstract: Many real-world multi-agent or multi-task evaluation scenarios can be naturally modelled as normal-form games due to inherent strategic (adversarial, cooperative, and mixed motive) interactions. These strategic interactions may be agentic (e.g. players trying to win), fundamental (e.g. cost vs quality), or complimentary (e.g. niche finding and specialization). In such a formulation, it is the strategies (actions, policies, agents, models, tasks, prompts, etc.) that are rated. However, the rating problem is complicated by redundancy and complexity of N-player strategic interactions. Repeated or similar strategies can distort ratings for those that counter or complement them. Previous work proposed ``clone-invariant'' ratings to handle such redundancies, but this was limited to two-player zero-sum (i.e. strictly competitive) interactions. This work introduces the first N-player general-sum clone-invariant rating, called \emph{deviation ratings}, based on coarse correlated equilibria. The rating is explored on several domains including LLMs evaluation.

Primary Area: other topics in machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3888