Chart Question Answering based on Modality Conversion and Large Language Models

Yi-Cheng Liu, Wei-Ta Chu

Abstract: A two-stage chart question answering system is proposed in this paper. Chart/plot images are first converted into structured text-based data by a transformer-based conversion model. Based on the structured text data, a large language model (LLM) is employed to answer the given questions to achieve chart-related question answering. Techniques like chain-of-thoughts, self-consistency, and program of thoughts are utilized to prompt the LLM based on the one-shot learning scheme. We also found that, by rephrasing questions several times and asking the LLM, different answers may be obtained. Aggregating these answers gives rise to performance gain. Overall, we show the proposed method is competitive or even better than the state of the arts, with smaller model size and requiring less training data.

External IDs:doi:10.1145/3643479.3662057