MME-FINANCE: A Multimodal Finance Benchmark for Expert-level Understanding and Reasoning

Ziliang Gan, Yu Lu, Dong Zhang, Haohan Li, Che Liu, Jian Liu, Ji Liu, Haipang WU, Chaoyou Fu, Zenglin Xu, Rongjunchen Zhang, Yong Dai

Keywords: Multimodal; Benchmark

Abstract:

The remarkable capability of existing Multimodal Large Language Models~(MLLMs) to understand general natural images have been extensively demonstrated in plentiful benchmarks. Nevertheless, the potential of MLLMs in finance domain remains to be fully explored. Financial images exhibit a wide range of variations, encompass intricate details, and demand professional expertise for proper interpretation, thereby posing a significant challenge for MLLMs in terms of their fine-grained perception and complex reasoning capabilities. To bridge this gap, we introduce MME-FINANCE, a novel benchmark designed specifically to assess MLLMs' performance in open-ended financial Visual Question Answering (VQA). Our benchmark consists of over 1,000 VQA pairs spanning a wide range of complex financial scenarios. We devise multi-tiered financial tasks tailored to the specific characteristics of the financial domain, aiming to comprehensively evaluate the perception, reasoning, and cognition capabilities of MLLMs.
Furthermore, we employ a multimodal evaluation approach that incorporates visual data to score the model predictions, thereby aligning more closely with human judgment. Extensive experimental evaluations of 18 mainstream MLLMs reveal their limitations in financial tasks and provide insights to inspire further research.

Primary Area: datasets and benchmarks

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5673