Efficient Continue Training of Temporal Language Model with Structural Information
Submission Type: Regular Long Paper
Submission Track: Language Modeling and Analysis of Language Models
Submission Track 2: Language Modeling and Analysis of Language Models
Keywords: Temporal Generalization, Syntactic Change, Temporal Language Model, Pre-trained Language Model
Abstract: Current language models are mainly trained on
snap-shots of data gathered at a particular time, which decreases their capability to generalize over time and model language change.
To model the \textit{time} variable, existing works have explored temporal language models (e.g., TempoBERT) by directly incorporating the timestamp into the training process. While effective to some extent, these methods are limited by the superficial temporal information brought by timestamps, which fails to learn the inherent changes of linguistic components. In this paper, we empirically confirm that the performance of pre-trained language models (PLMs) is closely affiliated with syntactically changed tokens. Based on this observation, we propose a simple yet effective method named \textit{\textbf{S}yntax-\textbf{G}uided \textbf{T}emporal \textbf{L}anguage \textbf{M}odel}~(SG-TLM), which could learn the inherent language changes by capturing an intrinsic relationship between the \textit{time} prefix and the tokens with salient syntactic change.
Experiments on two datasets and three tasks demonstrate that our model outperforms existing temporal language models in both memorization and generalization capabilities.
Extensive results further confirm the effectiveness of our approach across different model frameworks, including both encoder-only and decoder-only models (e.g., LLaMA).
Our code is available at \url{https://github.com/zhaochen0110/TempoLM}.
Submission Number: 5264
Loading