From Data to Insights: Exploring Program-of-Thoughts Prompting for Chart Summarization

ACL ARR 2025 July Submission519 Authors

Abstract: Charts play a critical role in conveying numerical data insights through structured visual representations. However, semantic visual understanding and numerical reasoning requirements hinder the accurate description of charts, interpreting a challenging task in chart summarization. Despite recent advancements in visual language models (VLMs), approaches lack robust mechanisms for verifying statistical fact correctness and are computationally heavy. To address this gap, this paper explores a strategy of using zero-shot learning to motivate the lightweight VLMs to perform computational reasoning, via Python programs as intermediaries to derive valid summary statistics for chart understanding. Specifically, we introduce a novel chart-to-dictionary auxiliary task, offering a more flexible representation compared to traditional chart-to-table methods, making it particularly well-suited for integration with the Program-of-Thought (PoT) strategy. Experimental results demonstrate our strategy performs on par with existing chart summarization methods across semantic and factual metrics. Code is available on https://anonymous.4open.science/r/ZeroShot-PoT-C2T-5A6B.

Paper Type: Long

Research Area: Language Modeling

Research Area Keywords: Language Modeling, Multimodality and Language Grounding to Vision, Robotics and Beyond, Summarization

Contribution Types: NLP engineering experiment, Approaches to low-resource settings, Data analysis

Languages Studied: English

Submission Number: 519