SKATE, a Scalable Tournament Eval: Weaker LLMs differentiate between stronger ones using verifiable challenges

Dewi Sid William Gould, Bruno Kacper Mlodozeniec, Samuel F. Brown

Keywords: automated evaluation, scalable evaluation

TL;DR: We introduce SKATE: a novel evaluation framework in which large language models (LLMs) compete by generating and solving verifiable tasks for one another.

Abstract:

Evaluating the capabilities and risks of frontier AI models is paramount, yet current methods demand extensive domain expertise, hindering their scalability as these models rapidly evolve. We introduce SKATE: a novel evaluation framework in which large language models (LLMs) compete by generating and solving verifiable tasks for one another. Our core insight is to treat evaluation as a game: models act as both task-setters and solvers, incentivized to create questions which highlight their own strengths while exposing others' weaknesses. SKATE offers several key advantages, balancing scalability, open-endedness, and objectivity. It is fully automated, data-free, and scalable, requiring no human input or domain expertise. By using verifiable tasks rather than LLM judges, scoring is objective. Unlike domain-limited programmatically-generated benchmarks (e.g. chess-playing or spatial reasoning), having LLMs creatively pose challenges enables open-ended and scalable evaluation. As a proof of concept, we introduce LLM-set code-output-prediction (COP) challenges as a verifiable and extensible framework in which to test our approach. Using a TrueSkill-based ranking system, we evaluate six frontier LLMs and find that:
(1) weaker models can score stronger ones consistently, reliably differentiating between them, and
(2) LLM-based systems are capable of self-preferencing behavior, generating questions that align with their own capabilities, and
(3) SKATE automatically surfaces fine-grained capability differences between models.
Our findings are an important step towards general, scalable evaluation frameworks which can keep pace with LLM progress.

Primary Area: datasets and benchmarks

Submission Number: 1728