Go to ICLR 2025 Conference homepageUnveiling Context-Aware Criteria in Self-Assessing LLMs
The use of large language models (LLMs) as evaluators has garnered significant attention due to their potential to rival human-level evaluations in long-form re- sponse assessments. However, current LLM evaluators rely heavily on static, human-defined criteria, limiting their ability to generalize across diverse gener- ative tasks and incorporate context-specific knowledge. In this paper, we pro- pose a novel Self-Assessing LLM framework that integrates Context-Aware Cri- teria (SALC) with dynamic knowledge tailored to each evaluation instance. This instance-level knowledge enhances the LLM evaluator’s performance by provid- ing relevant, context-aware insights that pinpoint the important criteria specific to the current instance. Additionally, the proposed framework adapts seamlessly to various tasks without relying on predefined human criteria, offering a more flex- ible evaluation approach. Empirical evaluations demonstrate that our approach significantly outperforms existing baseline evaluation frameworks, yielding im- provements ranging from 5% across a wide variety of datasets. Furthermore, by leveraging knowledge distillation techniques, we fine-tuned smaller language models for criteria generation and evaluation, achieving comparable or superior performance to larger models with much lower cost. Our method also exhibits a 5% improvement on the Alpaca leaderboard when employed for preference data generation in Direct Preference Optimization (DPO), underscoring its efficacy as a robust and scalable evaluation framework.