MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning

Anonymous

Abstract:

With the rapid development of large language models (LLMs) and their integration into large multimodal models (LMMs), there has been
impressive progress in zero-shot completion of user-oriented vision-language tasks. However, a gap remains in the domain of chart image
understanding due to the distinct abstract components in charts. To address this, we introduce a large-scale MultiModal Chart Instruction (MMC-Instruction) dataset comprising 600k instances supporting diverse tasks and chart types. Leveraging this data, we develop Multi-
Modal Chart Assistant (MMCA), an LMM that achieves state-of-the-art performance on existing chart QA benchmarks. Recognizing the
need for a comprehensive evaluation of LMM chart understanding, we also propose a Multi-Modal Chart Benchmark (MMC-Benchmark), a comprehensive human-annotated benchmark with 9 distinct tasks evaluating reasoning capabilities over charts. Extensive experiments
on MMC-Benchmark reveal the limitations of existing LMMs on correctly interpreting charts, even for the most recent GPT-4V model. Our
work provides an instruction-tuning methodology and benchmark to advance multimodal understanding of charts.

Paper Type: long

Research Area: Multimodality and Language Grounding to Vision, Robotics and Beyond

Contribution Types: Data resources

Languages Studied: English