Learning Disentangled Document Representations Based on a Classical Shallow Neural Encoder

Yuro Kanada, Sumio Fujita, Yoshiyuki Shoji

Abstract: This paper proposes a document embedding method designed to obtain disentangled distributed representations. The resulting representations are expected to satisfy two key criteria: independence across dimensions and semantic interpretability of each dimension. We enhanced a classic shallow neural network-based embedding model with two modifications: 1) guidance task integration, where the network is trained to perform both a simple auxiliary metadata prediction task and a surrounding term prediction task simultaneously, and 2) loss regularization for independence, where the loss function includes both prediction accuracy and the independence across dimensions (i.e., the Kullback-Leibler divergence from a multivariate normal distribution). We evaluated the proposed method through both automatic and human-subject experiments using synthetic datasets and movie review texts. Experimental results show that even shallow neural networks can generate disentangled representations when dimensional independence is explicitly promoted.

External IDs:doi:10.1007/978-3-032-11976-6_5