Continual Learning Knowledge Graph Embeddings for Dynamic Knowledge Graphs

Zongsheng Cao

Keywords: Continual Learning, Dynamic Knowledge Graphs

Abstract: Knowledge graphs (KG) have shown great power in representing the facts for numerous downstream applications. Notice that the KGs are usually evolving and growing with the development of the real world, due to the change of old knowledge and the emergence of new knowledge, thus the study of dynamic knowledge graphs attracts a new wave. However, conventional work mainly pays attention to learning new knowledge based on existing knowledge while neglecting new knowledge and old knowledge should contribute to each other. Under this circumstance, they cannot tackle the following two challenges: (C1) transfer the knowledge from the old to the new without retraining the entire KG; (C2) alleviate the catastrophic forgetting of old knowledge with new knowledge. To address these issues, we revisit the embedding paradigm for dynamic knowledge graphs and propose a new method termed \textbf{C}ontinual \textbf{L}earning \textbf{K}nowledge \textbf{G}raph \textbf{E}mbeddings (\textbf{CLKGE}). In this paper, we establish a new framework, allowing new and old knowledge to be gained from each other. Specifically, to tackle the (C1), we leverage continual learning to conduct the knowledge transfer and obtain new knowledge based on the old knowledge graph. In the face of (C2), we utilize the energy-based model, learning an energy manifold for the knowledge representations and aligning new knowledge and old knowledge such that their energy on the manifold is minimized, hence can alleviate catastrophic forgetting with the assistance of new knowledge. On top of this, we propose a theoretical guarantee that our model can converge to the optimal solution for the dynamic knowledge graphs. Moreover, we conduct extensive experimental results demonstrating that CLKGE achieves state-of-the-art performance compared with the embedding baselines.

Primary Area: learning on graphs and other geometries & topologies

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4417