Natural Language Generation Meets Data Visualization: Vis-to-Text and its Duality with Text-to-Vis

Yuanfeng Song, Xiaoling Huang, Xuefang Zhao, Raymond Chi-Wing Wong

Abstract: Data visualizations (DVs) refer to the methodologies and tools where visual elements like charts, bars and scatters are used to convey summaries behind the raw data. However, it usually takes much effort to master DV, even for data scientists and experts, not to mention beginners. Hence, a popular task named Text-to-Vis, focusing on automatically generating data visualizations (DVs) from natural language questions (NLQs), has been introduced and has recently been gaining great attention from both the database and the data mining communities. In this paper, we propose the reversed task Vis-to-Text, which aims to generate human-readable descriptions to explain complicated DVs for educational purposes. Intuitively, text-to-vis and vis-to-text are strongly correlated to one another, i.e., the input of text-to-vis is the output of vis-to-text, and vice versa. This relationship is generally known as duality and has been validated to be important for improving the performances of both tasks in machine translation, question answering, and dialogue systems. However, its effectiveness in text-to-vis and vis-to-text is under-explored. In this paper, we make use of the duality to optimize both tasks. We first design a Transformer-based network to tackle the vis-to-text task. Then, we further explore a dual training framework to simultaneously optimize the two tasks. More specifically, we analyze the duality and finally convert it into a corresponding regularization term to constrain the loss function to guide the model training process. Finally, we evaluate our approach on a public dataset, and the experimental results validate the rationale of this new proposed vis-to-text task and also show that this dual framework can boost the performance of the vis-to-text task over existing baselines.