JudgeAgent: Beyond Static Benchmarks for Knowledge-Driven and Dynamic LLM Evaluation

ACL ARR 2026 January Submission3402 Authors

Keywords: large language model, LLM agents, automatic evaluation

Abstract:

Current evaluation methods for large language models (LLMs) primarily rely on static benchmarks, presenting two major challenges: limited knowledge coverage and fixed difficulties that mismatch with the evaluated LLMs.
These limitations lead to superficial assessments of LLM knowledge, thereby impeding the targeted model optimizations.
To bridge this gap, we propose JudgeAgent, a knowledge-driven and dynamic evaluation framework for LLMs.
To address the challenge of limited knowledge coverage, JudgeAgent leverages LLM agents equipped with context graphs to traverse knowledge structures systematically for question generation.
Furthermore, to mitigate data contamination and difficulty mismatch, it adopts a difficulty-adaptive and multi-turn interview mechanism.
Thereby, JudgeAgent can achieve comprehensive evaluations and facilitate more effective improvement of LLMs.
Empirical results demonstrate that JudgeAgent enables more comprehensive evaluations and facilitates effective model iterations, highlighting the potential of this knowledge-driven and dynamic evaluation paradigm.
The source code is available on https://anonymous.4open.science/r/JudgeAgent.

Paper Type: Long

Research Area: Resources and Evaluation

Research Area Keywords: automatic evaluation,automatic evaluation

Contribution Types: NLP engineering experiment

Languages Studied: English

Submission Number: 3402