Curriculum-Guided Self-Supervised Representation Learning of Dynamic Heterogeneous Networks

Namgyu Jung, David Camacho, Chang Choi, O-Joun Lee

Abstract: Since most real-world network data include nodes and edges that evolve gradually, an embedding model for dynamic heterogeneous networks is crucial for network analysis. Transformer models have remarkable success in natural language processing but are rarely applied to learning representations of dynamic heterogeneous networks. In this study, we propose a new transformer model (DHG-BERT) that (i) constructs a dataset based on a network curriculum and (ii) includes pre/post-learning through self-supervised learning. Our proposed model learns complex relationships by leveraging an easier understanding of relationships through data reconstruction. Additionally, we use self-supervised learning to learn network structural features and temporal changes in structure and then fine-tune the proposed model by focusing on specific meta-paths by considering domain characteristics or target tasks. We evaluated the quality of the vector representation produced by the proposed transducer model using real bibliographic networks. Our model achieved an average accuracy of 0.94 in predicting research collaboration between researchers, outperforming existing models by a minimum of 0.13 and a maximum of 0.35. As a result, we confirmed that DHG-BERT is an effective transformer model tailored to dynamic heterogeneous network embeddings. Our study highlights the model’s ability to understand complex network relationships and appropriately capture the structural nuances and temporal changes inherent in networks. This study provides future research directions for applying the transformer model to real-world network data and a new approach to analyzing dynamic heterogeneous networks using transformers.