MAC: A Multimodal Benchmark for Understanding and Generating Academic Journal Covers
Keywords: Benchmark, Multi-modality, Large Multimodal Models
Abstract: We introduce the Multimodal Academic Cover (MAC) benchmark to address the challenges of Large Multimodal Models (LMMs) in understanding and generating academic journal covers. While LMMs have demonstrated significant progress in creative arts and everyday applications, their capabilities in comprehending complex academic visuals and narratives remain underexplored. MAC comprises
a collection of 5,872 cover images, accompanying cover stories, and associated articles from 40 prominent academic journals, providing a rich dataset for evaluation. We design bidirectional generative tasks—Image2Text and Text2Imag to assess authenticity and creativity in generating cover images and stories. Current LMMs, including DALL·E 3, GPT-4V, Gemini, CogView-3, GLM-4V, LLaVA, LLaMA-adapter, and MiniGPT4, are evaluated on this benchmark. Furthermore, we propose Multimodal Agent Linkage (MAL), a novel method to enhance conceptual comprehension within a long-context window. In-context learning techniques, such as few-shot learning, are also explored to improve the effectiveness of LMMs. All benchmarks, prompts, and codes will be released publicly.
Primary Area: datasets and benchmarks
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 10373
Loading