A Spectral Viewpoint on Continual Relation Extraction

Huy Huu Nguyen, Chien Van Nguyen, Linh Van Ngo, Anh Tuan Luu, Thien Huu Nguyen

Submission Type: Regular Short Paper

Submission Track: Information Extraction

Keywords: Relation Extraction, Information Extraction, Continual Learning, Spectral Analysis

TL;DR: We present a novel spectral analysis and regularization method for the catastrophic forgetting issue in continual learning of relation extraction.

Abstract: Continual Relation Extraction (CRE) aims to continuously train a model to learn new relations while preserving its ability on previously learned relations. Similar to other continual learning problems, in CRE, models experience representation shift, where learned deep space changes in the continual learning process, which leads to the downgrade in the performance of the old tasks. In this work, we will provide an insight into this phenomenon under the spectral viewpoint. Our key argument is that, for each class shape, if its eigenvectors (or spectral components) do not change much, the shape is well-preserved. We then conduct a spectral experiment and show that, for the shape of each class, the eigenvectors with larger eigenvalue are more preserved after learning new tasks which means these vectors are good at keeping class shapes. Based on this analysis, we propose a simple yet effective class-wise regularization that improve the eigenvalues in the representation learning. We observe that our proposed regularization leads to an increase in the eigenvalues. Extensive experiments on two benchmark datasets, FewRel and TACRED, show the effectiveness of our proposed method with significant improvement in performance compared to the state-of-the-art models. Further analyses also verify our hypothesis that larger eigenvalues lead to better performance and vice versa.

Submission Number: 5335