On the Evaluation of Methods for Temporal Knowledge Graph Forecasting

Julia Gastinger, Timo Sztyler, Lokesh Sharma, Anett Schuelke

Keywords: Temporal Knowledge Graphs, Temporal Knowledge Graph Forecasting, Temporal Knowledge Graph Extrapolation, Benchmarking

TL;DR: The paper discusses problems and inconsistencies with evaluation settings in Temporal Knowledge Graph (TKG) Forecasting, forms a unified evaluation protocol, and re-evaluates state-of-the-art TKG Forecasting models on this evaluation protocol.

Abstract: Due to its ability to incorporate and leverage time information in relational data, Temporal Knowledge Graph (TKG) learning has become an increasingly studied research field. With the goal of predicting the future, researchers have presented innovative methods for what is called Temporal Knowledge Graph Forecasting. However, the experimental procedures in this line of work show inconsistencies that strongly influence empirical results and thus lead to distorted comparisons among models. This work focuses on the evaluation of TKG Forecasting models: we describe evaluation settings commonly used in this research area and shed light on its scholarship issues. Further, we provide a unified evaluation protocol and carry out a re-evaluation of state-of-the-art models on the most common datasets under such a setting. Finally, we show the difference in results caused by different evaluation settings. We believe that this work provides a solid foundation for future evaluations of TKG Forecasting models and can thus contribute to the development of this growing research area.

Paper Format: full paper (8 pages)