Report Cards: Qualitative Evaluation of LLMs Using Natural Language Summaries

Blair Yang, Fuyang Cui, Keiran Paster, Jimmy Ba, Pashootan Vaezipoor, Silviu Pitis, Michael R. Zhang

Track: Technical

Keywords: LLM eval, Auto evaluation, Intrepretability

TL;DR: We introduce Report Cards, interpretable summaries that more holistically evaluate LLMs than traditional benchmarks.

Abstract: The generality and dynamic nature of large language models (LLMs) make it difficult for conventional quantitative benchmarks to accurately assess their capabilities. We propose Report Cards, which are human-interpretable, natural language summaries of model behavior for specific skills or topics. We develop a framework to evaluate Report Cards based on three criteria: specificity (ability to distinguish between models), faithfulness (accurate representation of model capabilities), and interpretability (clarity and relevance to humans). We also propose an iterative algorithm for generating Report Cards without human supervision. Through experimentation with popular LLMs, we demonstrate that Report Cards provide insights beyond traditional benchmarks and can help address the need for a more interpretable and holistic evaluation of LLMs.

Submission Number: 67