League: Leaderboard Generation on Demand

Jian Wu, Jiayu Zhang, Luodan Zhang, Dongyuan Li, Linyi Yang, Aoxiao Zhong, Renhe Jiang, Qingsong Wen, Yue Zhang

Keywords: Leaderboard Generation, NLP Application

Abstract:

Leaderboard gathers experimental results from various sources into unified rankings, giving researchers clear standards for measuring progress while facilitating fair comparisons. However, with thousands of academic papers updated daily, manually tracking each paper's methods, results, and experimental settings has become impractical, creating an urgent need for automated leaderboard generation.
Although large language models offer promise in automating this process, challenges such as multi-document summarization, fair result extraction, and consistent experimental comparison remain underexplored. To address these challenges, we introduce Leaderboard Auto Generation (League), a novel and well-organized framework for automatic generation of leaderboards on a given research topic in rapidly evolving fields like Artificial Intelligence. League employs a systematic pipeline encompassing paper collection, result extraction and integration, leaderboard generation, and quality evaluation. Through extensive experiments across multiple research domains, we demonstrate that League produces leaderboards comparable to manual curation while significantly reducing human effort.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 12114