JMMMU-Pro: Text-embedded Japanese Multi-discipline Multimodal Understanding Benchmark via Vibe Benchmark Construction

ACL ARR 2026 January Submission2144 Authors

Keywords: benchmark, multilingual, Japanese, culture-aware benchmark, large multimodal model, image generation model

Abstract: This paper introduces JMMMU-Pro, a text-embedded Japanese Multi-discipline Multimodal Understanding Benchmark, and Vibe Benchmark Construction, a scalable construction method. Following the evolution from MMMU to MMMU-Pro, JMMMU-Pro extends JMMMU by composing the question image and question text into a single image, thereby creating a benchmark that requires integrated visual-textual understanding through visual perception. To build JMMMU-Pro, we propose Vibe Benchmark Construction, a methodology in which an image generative model (e.g., Nano Banana Pro) produces candidate visual questions, and humans verify the outputs and, when necessary, regenerate with adjusted prompts to ensure quality. By leveraging Nano Banana Pro's highly realistic image generation capabilities and its ability to embed clean Japanese text, we construct a high-quality benchmark at low cost, covering a wide range of background and layout designs. Experimental results show that all open-source LMMs struggle substantially with JMMMU-Pro, underscoring JMMMU-Pro as an important benchmark for guiding future efforts in the open-source community. We believe that JMMMU-Pro provides a more rigorous evaluation tool for assessing the Japanese capabilities of LMMs and that our Vibe Benchmark Construction also offers an efficient guideline for future development of text-embedded VQA benchmarks.

Paper Type: Long

Research Area: Multilinguality and Language Diversity

Research Area Keywords: benchmark, multilingual, Japanese, culture-aware benchmark, large multimodal model

Contribution Types: Data resources

Languages Studied: Japanese

Submission Number: 2144