Go to ACL ARR 2024 December homepageATLAS: A Reasoning-Guided Attribution Framework for Mathematical Chart Analysis
Abstract: The human-like capability of Multimodal Large Language Models (MLLMs) like GPT-4o to process both text and images enables them to help humans with quantitative analysis of charts. However, these models are known to hallucinate, more so on vision language tasks; our initial study on a sample from the ChartQA dataset \cite{masry-etal-2022-chartqa} indicates that GPT-4o provides accurate answers only $58\%$ of the time for questions on chart images. In this paper, we introduce attribution for chart-based mathematical questions, where bounding boxes identify the key regions that justify answers, building on recent work in factual verification for text-based question answering. Taking inspiration from Chain-of-Thought (CoT)-like prompting strategies, we hypothesize that understanding step-by-step reasoning can help in improving attribution accuracy in chart-based mathematical question-answering. We propose a semi-automatic approach to obtain a benchmarking dataset comprising 7,819 diverse samples with charts, questions, reasoning steps, and attribution annotations. We introduce a method using the open-source Internlm-XComposer2 model with Partial Low-Rank Adaptation, treating vision and language tokens equally to generate high-quality attributions through detailed reasoning steps. Our experimental results show that our approach enhances attribution quality by $\sim$15\%, advancing the development of interpretable and trustworthy chart-based AI systems.
Paper Type: Long
Research Area: Multimodality and Language Grounding to Vision, Robotics and Beyond
Research Area Keywords: multimodality, vision question answering, image text matching
Contribution Types: Model analysis & interpretability, NLP engineering experiment, Data analysis, Theory
Languages Studied: English
Submission Number: 1984
Loading