From Data to Insight: Exploring Program-of-Thoughts Prompting for Chart Summarization

ACL ARR 2025 May Submission4746 Authors

Abstract: Charts play a critical role in conveying numerical data insights through structured visual representations. However, semantic visual understanding and numerical reasoning requirements hinder the accurate description of charts, interpreting a challenging task in chart summarization. Despite recent advancements in visual language models (VLMs), approaches lack robust mechanisms for verifying statistical fact correctness and are computationally heavy. To address this gap, this paper explores the potential of using zero-shot learning to motivate the lightweight VLMs to perform computational reasoning via Python programs as intermediate outputs to derive summary statistics valid for chart understanding. Specifically, we introduce a novel chart-to-dictionary auxiliary task, offering a more flexible representation compared to traditional chart-to-table methods, making it particularly well-suited for integration with the Program-of-Thought (PoT) strategy. Experimental results demonstrate that our method performs on par with existing chart summarization methods across machine translation and text generation metrics. We release the code at the GitHub link.

Paper Type: Long

Research Area: Language Modeling

Research Area Keywords: Language Modeling, Multimodality and Language Grounding to Vision, Robotics and Beyond, Summarization

Contribution Types: NLP engineering experiment, Approaches to low-resource settings, Data analysis

Languages Studied: English

Submission Number: 4746