TextGraphBART: Unifying Graph and Text with Structure Token

Ching-Wen Cheng, PING-CHENG YEH

Keywords: transformer, text generation, graph generation, text-to-graph, graph-to-text

TL;DR: We propose a novel encoding method called ``Structure Token'' to unify the processing and generation of both graphs and texts with a single transformer-based model.

Abstract: We propose a novel encoding method called ``Structure Token'' to unify the processing and generation of both graphs and texts with a single transformer-based model. This method allows graphs with text labels to be generated by a series of tokens, enabling both graph and text data to be handled interchangeably. By utilizing structure tokens, our model learns a unified representation, enhancing the ability to process diverse data without requiring extra modules or models. Additionally, the model can be trained like most transformer models with simply cross-entropy loss. To demonstrate the effectiveness of our method, we introduce a pre-training scheme inspired by mBART but adapted to leverage structure tokens. Our model, named TextGraphBART, uses the same architecture as normal Transformer Encoder-Decoder models with small modifications on the input and output to accommodate structure tokens. The evaluations show that this approach achieves comparable results against baseline models of similar sizes on both text-to-graph and graph-to-text generation tasks, without needing specialized loss functions or sampling techniques. These findings suggest that our approach can effectively bridge the gap between textual and structural data representations, and the design of encoding method could offer a new direction for future improvement.

Primary Area: Generative models

Submission Number: 14404