Information Theoretic Regularization for Learning Global Features by Sequential VAE

Kei Akuzawa, Yusuke Iwasawa, Yutaka Matsuo

Keywords: variational autoencoder, VAE, disentanglement, global features, sequential models, representation learning, mutual information

Abstract: Sequential variational autoencoders (VAEs) with global latent variable $z$ have been studied for the purpose of disentangling the global features of data, which is useful in many downstream tasks. To assist the sequential VAEs further in obtaining meaningful $z$, an auxiliary loss that maximizes the mutual information (MI) between the observation and $z$ is often employed. However, by analyzing the sequential VAEs from the information theoretic perspective, we can claim that simply maximizing the MI encourages the latent variables to have redundant information and prevents the disentanglement of global and local features. Based on this analysis, we derive a novel regularization method that makes $z$ informative while encouraging the disentanglement. Specifically, the proposed method removes redundant information by minimizing the MI between $z$ and the local features by using adversarial training. In the experiments, we trained state-space and autoregressive model variants using speech and image datasets. The results indicate that the proposed method improves the performance of the downstream classification and data generation tasks, thereby supporting our information theoretic perspective in the learning of global representations.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: To assist sequential VAEs with global latent variable, we propose a new information theoretic regularization method for disentangling the global factors.

Reviewed Version (pdf): https://openreview.net/references/pdf?id=BFiXxDDRfD