Representation Topology Divergence: A Method for Comparing Neural Network Representations.

Serguei Barannikov, Ilya Trofimov, Nikita Balabin, Evgeny Burnaev

Keywords: representation learning, understanding deep learning, topological data analysis

Abstract: Comparison of data representations is a complex multi-aspect problem that has not enjoyed a complete solution yet. We propose a method for comparing two data representations. We introduce the Representation Topology Divergence (RTD) score measuring the dissimilarity in multi-scale topology between two point clouds of equal size with a one-to-one correspondence between points. The data point clouds are allowed to lie in different ambient spaces. The RTD score is one of the few TDA-based practical methods applicable to real machine learning datasets. Experiments show the agreement of RTD with the intuitive assessment of data representation similarity. The proposed RTD score is sensitive to the data representation's fine topological structure. We use the RTD score to gain insights on neural networks representations in computer vision and NLP domains for various problems: training dynamics analysis, data distribution shift, transfer learning, ensemble learning, disentanglement assessment.

One-sentence Summary: We introduce the Representation Topology Divergence (RTD) score measuring the dissimilarity of neural representations, that is sensitive to the data representation's fine topological structure