Unsupervised Learning of Equivariant Structure from Sequences

Takeru Miyato, Masanori Koyama, Kenji Fukumizu

Keywords: Disentanglement, Symmetry, Unsupervised Learning, Equivariance

Abstract:

In this study, we present \textit{meta-sequential prediction} (MSP), an unsupervised framework to learn the symmetry from the time sequence of length at least three.
Our method leverages the stationary property~(e.g. constant velocity, constant acceleration) of the time sequence to learn the underlying equivariant structure of the dataset by simply training the encoder-decoder model to be able to predict the future observations.
We will demonstrate that, with our framework, the hidden disentangled structure of the dataset naturally emerges as a by-product by applying \textit{simultaneous block-diagonalization} to the transition operators in the latent space, the procedure which is commonly used in representation theory to decompose the feature-space based on the type of response to group actions.
We will showcase our method from both empirical and theoretical perspectives.
Our result suggests that finding a simple structured relation and learning a model with extrapolation capability are two sides of the same coin. The code is available at https://github.com/takerum/meta_sequential_prediction.

TL;DR: We show that equivariance can be learned from a set of sequences with different but constant accelerations/velocities, and show that disentanglement emerges when we train the model to be able to linearly predict the future in the latent space.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2210.05972/code)