ChartAlignBench: A Benchmark for Chart Grounding & Dense Alignment

Aniruddh Bansal, Davit Soselia, Dang Nguyen, Tianyi Zhou

Keywords: benchmarking, chart understanding, VLMs, multi chart alignment, fine grained visual grounding

TL;DR: A benchmark and evaluation framework for vision–language models on fine-grained chart grounding and multi-chart alignment with augmented downstream QA reasoning.

Abstract: Charts play important roles in visualization, reasoning, and communication in data analysis and idea exchange between humans. However, vision-language models (VLMs) still lack accurate understanding of the details and struggle to extract fine-grained structural information from charts. Such limitations in chart grounding also hinder their capability to compare multiple charts and reason about their difference. In this paper, we develop a novel "**ChartA**lign **B**enchmark (*ChartAB*)" to provide a full-spectrum evaluation of VLMs in chart grounding tasks, i.e., extracting tabular data, allocating visualization elements, and recognizing various attributes from charts of diverse types and complexities. We develop a JSON template to facilitate the calculation of evaluation metrics specifically designed for each grounding task. By applying a novel two-stage inference workflow, the benchmark can further evaluate VLMs' capability of aligning and comparing elements/attributes in two charts. Our analysis of evaluations on several recent VLMs sheds novel insights on their perception biases, weaknesses, robustness, and hallucinations in chart understanding.

Primary Area: datasets and benchmarks

Submission Number: 9606