ConceptPsy: A comprehensive benchmark suite for hierarchical psychological concept understanding in LLMs

Junlei Zhang, Hongliang He, Lizhi Ma, Nirui Song, Shuyuan He, Shuai Zhang, Huachuan Qiu, Zhanchao Zhou, Anqi Li, Yong Dai, Renjun Xu, Zhenzhong Lan

Abstract: Highlights•Propose a metric to quantify low concept coverage in Chinese MMLU benchmarks.•Propose a conceptually comprehensive college-level psychological benchmark.•Propose using concept-wise labels for fine-grained model evaluation.•Fine-grained results reveal weaknesses of models on specific concepts.•Fine-grained results offer insights beyond subject-level, aiding model refinement.