Language Modeling with Graph Temporal Convolutional Networks

Hongyin Luo, Yichen Li, Jie Fu, James Glass

Abstract:

Recently, there have been some attempts to use non-recurrent neural models for language modeling.
However, a noticeable performance gap still remains.
We propose a non-recurrent neural language model, dubbed graph temporal convolutional network (GTCN), that relies on graph neural network blocks and convolution operations. While the standard recurrent neural network language models encode sentences sequentially without modeling higher-level structural information, our model regards sentences as graphs and processes input words within a message propagation framework, aiming to learn better syntactic information by inferring skip-word connections. Specifically, the graph network blocks operate in parallel and learn the underlying graph structures in sentences without any additional annotation pertaining to structure knowledge. Experiments demonstrate that the model without recurrence can achieve comparable perplexity results in language modeling tasks and successfully learn syntactic information.

Keywords: Graph Neural Network, Language Modeling, Convolution