An Information Theoretic Perspective on Disentangled Representation Learning

Xiaojiang Yang, Wendong Bi, Yu Cheng, Junchi Yan

Abstract: Existing works on disentangled representation learning usually lie on a common assumption: all factors in a disentangled representation should be independent. We argue that this assumption is not sufficient and another assumption is vital for disentangled representation learning: information contained in each factor of a disentangled representation is irrelevant to others, i.e. the containing information about data of factors is isolated. We formulate this assumption into two equivalent equations via mutual information, and theoretically show its relation with independence and conditional independence of factors in a representation. Meanwhile, we prove that conditional independence is satisfied in encoders of VAEs due to ``no-sharing-parameter block" and reparameterization trick. To highlight the importance of the proposed assumption, we show in experiments that violating the assumption leads to decline of disentanglement. Based on this assumption, we further propose to split the deeper layers in encoder to ensure parameters in these layers are not shared for different factors. The proposed encoder, called \textit{Split Encoder}, can be applied into other models and shows significant improvement in unsupervised learning of disentangled representations and reconstructions.

Code: https://github.com/website-for-iclr/our-dlib