Evaluation Metrics for Graph Generative Models: Problems, Pitfalls, and Practical SolutionsDownload PDF

29 Sept 2021, 00:31 (modified: 16 Mar 2022, 15:16)ICLR 2022 SpotlightReaders: Everyone
Keywords: graph generative models, model evaluation
Abstract: Graph generative models are a highly active branch of machine learning. Given the steady development of new models of ever-increasing complexity, it is necessary to provide a principled way to evaluate and compare them. In this paper, we enumerate the desirable criteria for such a comparison metric and provide an overview of the status quo of graph generative model comparison in use today, which predominantly relies on the maximum mean discrepancy (MMD). We perform a systematic evaluation of MMD in the context of graph generative model comparison, highlighting some of the challenges and pitfalls researchers inadvertently may encounter. After conducting a thorough analysis of the behaviour of MMD on synthetically-generated perturbed graphs as well as on recently-proposed graph generative models, we are able to provide a suitable procedure to mitigate these challenges and pitfalls. We aggregate our findings into a list of practical recommendations for researchers to use when evaluating graph generative models.
One-sentence Summary: We investigate the potential pitfalls of using MMD to evaluate graph generative models and propose recommendations for the practitioner on how to mitigate those challenges.
12 Replies