Go to ACL ARR 2025 February homepageProBench: Judging Multimodal Foundation Models on Open-ended Multi-domain Expert Tasks
Abstract: Solving expert-level multimodal tasks is a key milestone in general intelligence. As the capabilities of multimodal large language models (MLLMs) continue to evolve, evaluation of frontier multimodal intelligence becomes necessary yet challenging. In this work, we introduce ProBench, a benchmark of open-ended user queries encapsulating professional expertise and advanced reasoning. ProBench consists of 4,000 high-quality samples independently collected from professionals based on their productivity demands. It spans across 10 fields and 56 sub-fields, including science, arts, humanities, coding, mathematics, and creative writing. Experimentally, we evaluate and compare 24 latest models using MLLM-as-a-Judge. Our results reveal that although the best open-source models rival the proprietary ones, they all face significant challenges in visual perception, textual understanding, domain knowledge, and advanced reasoning. Our benchmark is publicly accessible at \url{TBC}.
Paper Type: Long
Research Area: Multimodality and Language Grounding to Vision, Robotics and Beyond
Research Area Keywords: multimodality
Contribution Types: Model analysis & interpretability, Data resources, Data analysis
Languages Studied: English, Portuguese, Italian, Vietnamese, Russian, Persian, Catalan, Indonesian, Dutch, Turkish, Polish, Chinese, Arabic, Danish, French, Spanish, German
Submission Number: 2885
Loading