BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks

Anna Sokol, Elizabeth M. Daly, Michael Hind, David Piorkowski, Xiangliang Zhang, Nuno Moniz, Nitesh V Chawla

Keywords: large language models, benchmarks, documentation, standardization, evaluation, transparency, BenchmarkCards

TL;DR: BenchmarkCards provides standardized documentation for large language model benchmarks, simplifying benchmark selection and usage.

Abstract:

Large language models (LLMs) are powerful tools capable of handling diverse tasks. Comparing and selecting appropriate LLMs for specific tasks requires systematic evaluation methods, as models exhibit varying capabilities across different domains. However, finding suitable benchmarks is difficult given the many available options. This complexity not only increases the risk of benchmark misuse and misinterpretation but also demands substantial effort from LLM users, seeking the most suitable benchmarks for their specific needs. To address these issues, we introduce BenchmarkCards, an intuitive and validated documentation framework that standardizes critical benchmark attributes such as objectives, methodologies, data sources, and limitations. Through user studies involving benchmark creators and users, we show that BenchmarkCards can simplify benchmark selection and enhance transparency, facilitating informed decision-making in evaluating LLMs.

Data & Code:
github.com/SokolAnn/BenchmarkCards
huggingface.co/datasets/ASokol/BenchmarkCards

Code URL: https://github.com/SokolAnn/BenchmarkCards

Primary Area: Datasets & Benchmarks for applications in language modeling and vision language modeling

Submission Number: 2225