Localized Generations with Deep Neural Networks for Multi-Scale Structured Datasets

Yoshihiro Nagano, Shiro Takagi, Yuki Yoshida, Masato Okada

Keywords: Variational autoencoder, Local learning, Model-agnostic meta-learning, Disentangled representation

TL;DR: Generalized Variational Autoencoder to be applicable to the dataset with local structure while keeping to avoid a heavy computation by the meta-learning with structural similarity assumption.

Abstract: Extracting the hidden structure of the external environment is an essential component of intelligent agents and human learning. The real-world datasets that we are interested in are often characterized by the locality: the change in the structural relationship between the data points depending on location in observation space. The local learning approach extracts semantic representations for these datasets by training the embedding model from scratch for each local neighborhood, respectively. However, this approach is only limited to use with a simple model, since the complex model, including deep neural networks, requires a massive amount of data and extended training time. In this study, we overcome this trade-off based on the insight that the real-world dataset often shares some structural similarity between each neighborhood. We propose to utilize the embedding model for the other local structure as a weak form of supervision. Our proposed model, the Local VAE, generalize the Variational Autoencoder to have the different model parameters for each local subset and train these local parameters by the gradient-based meta-learning. Our experimental results showed that the Local VAE succeeded in learning the semantic representations for the dataset with local structure, including the 3D Shapes Dataset, and generated high-quality images.

Code: https://drive.google.com/file/d/1vHfEt5RGrlLM77Ae0nCkRjIaQFtTqzMM/view