WenMind: A Comprehensive Benchmark for Evaluating Large Language Models in Chinese Classical Literature and Language Arts

Jiahuan Cao, Yang Liu, Yongxin Shi, Kai Ding, Lianwen Jin

Keywords: Benchmark, Large Language Models, Classical Chinese Literature and Language Arts

TL;DR: We constructed a comprehensive benchamrk for evaluating large language models in classical Chinese literature and language arts.

Abstract:

Large Language Models (LLMs) have made significant advancements across numerous domains, but their capabilities in Chinese Classical Literature and Language Arts (CCLLA) remain largely unexplored due to the limited scope and tasks of existing benchmarks. To fill this gap, we propose WenMind, a comprehensive benchmark dedicated for evaluating LLMs in CCLLA. WenMind covers the sub-domains of Ancient Prose, Ancient Poetry, and Ancient Literary Culture, comprising 4,875 question-answer pairs, spanning 42 fine-grained tasks, 3 question formats, and 2 evaluation scenarios: domain-oriented and capability-oriented. Based on WenMind, we conduct a thorough evaluation of 31 representative LLMs, including general-purpose models and ancient Chinese LLMs. The results reveal that even the best-performing model, ERNIE-4.0, only achieves a total score of 64.3, indicating significant room for improvement of LLMs in the CCLLA domain. We also provide insights into the strengths and weaknesses of different LLMs and highlight the importance of pre-training data in achieving better results.
Overall, WenMind serves as a standardized and comprehensive baseline, providing valuable insights for future CCLLA research.
Our benchmark and related code are available at \url{https://github.com/SCUT-DLVCLab/WenMind}.

Flagged For Ethics Review: true

Submission Number: 1013