Disentanglement and Local Directions of Variance

Alexander Rakowski, Christoph Lippert

Abstract: Previous line of research on learning disentangled representations in an unsupervised setting focused on enforcing an uncorrelated posterior. These approaches have been shown both empirically and theoretically to be insufficient for guaranteeing disentangled representations. Recent works postulate that an implicit PCA-like behavior might explain why these models still tend to disentangle, exploiting the structure of variance in the datasets. Here we aim to further verify those hypotheses by conducting multiple analyses on existing benchmark datasets and models, focusing on the relation between the structure of variance induced by the ground-truth factors and properties of the learned representations. We quantify the effects of global and local directions of variance in the data on disentanglement performance using proposed measures and seem to find empirical evidence of a negative effect of local variance directions on disentanglement. We also invalidate the robustness of models with a global ordering of latent dimensions against the local vs. global discrepancies in the data.