ChartMaster: Advancing Chart-to-Code Generation with Real-World Charts and Chart Similarity Reinforcement Learning

Wentao Tan, Qiong Cao, Chao Xue, Yibing Zhan, Changxing Ding, Xiaodong He

Keywords: Chart-to-Code Generation; Reinforcement Learning

Abstract:

The chart-to-code generation task requires MLLMs to convert chart images into executable code. This task faces two main challenges: limited data diversity and the difficulty of maintaining visual consistency between generated charts and the original ones. Existing datasets mainly rely on synthetic seed data to prompt GPT models for code generation, resulting in homogeneous samples that limit model generalization to real-world chart styles. To address this, we propose **ReChartPrompt**, leveraging real-world, human-designed charts extracted from arXiv papers as prompts. By harnessing the rich content and diverse visual styles of arXiv charts, we construct ReChartPrompt-240K, a large-scale and highly diverse dataset that better reflects realistic chart variations.
For the second challenge, although SFT improves code understanding by optimizing next-token prediction, it does not provide direct supervision on visual features. As a result, it often fails to guarantee that the generated charts visually match the original ones. To address this, we propose **ChartSimRL**, a GRPO-based reinforcement learning algorithm guided by a novel chart similarity reward. This reward consists of two components: *attribute similarity*, which measures the overlap of chart attributes like layout and color between the generated and original charts, and *visual similarity*, which evaluates overall visual features, including texture, using convolutional neural networks. Unlike traditional text-based rewards, our reward accounts for the multimodal nature of the chart-to-code generation task, significantly enhancing the model's ability to accurately reproduce charts.
Integrating ReChartPrompt and ChartSimRL, we develop the **ChartMaster** model, achieving SOTA results among 7B-parameter models and rivaling GPT-4o on various chart-to-code benchmarks.
We will release all code, datasets, and models to facilitate further research.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 1258