FlagEval Findings Report: A Preliminary Evaluation of Large Reasoning Models on Automatically Verifiable Textual and Visual Questions

Bowen Qin, Chen Yue, Jin-Ge Yao, Jing-Shu Zheng, Miguel Hu Chen, richeng xuan, Shibei Meng, Tong-Shuai Ren, Xi Yang, Xuejing Li, Yesheng Liu, Yonghua Lin, Zheqi He, Zhongyuan Wang

Keywords: reasoning models, evaluation, benchmark, large language models, vision-language models, tool use, hallucination

TL;DR: We conduct a moderate-scale contamination-free (to some extent) evaluation of current large reasoning models (LRMs) with some preliminary findings.

Abstract: We conduct a moderate-scale contamination-free (to some extent) evaluation of current large reasoning models (LRMs) with some preliminary findings. We also release ROME, our evaluation benchmark for vision language models.

Submission Number: 214