One Stage Autoencoders for Multi-Domain Learning

Mohamed Zayan, Dina Khattab

Keywords: energy-based models, probabilistic models, autoencoders, optimization, learning representations, unsupervised learning

Abstract: Autoencoders (AEs) are widely being used for representation learning. Empirically AEs are capable of capturing hidden representations of a given domain precisely. However, in principle AEs’ latent representation might be misleading, especially in the presence of weak encoding constraints. In this paper, we introduce one stage autoencoders (OSAs) to induce searching for patterns while training artificial neural networks. We propose two different frameworks for OSAs; Autoclave Restricted Boltzmann Machines (ACRBMs) and Local Observer Convolution (LOC). Both frameworks are compatible with artificial neural networks and trained via direct backpropagation (end-to-end training). Furthermore, they are scalable and require significantly less number of parameters than traditional AEs. ACRBMs are extensions of RBMs that are able to describe a given domain symmetrically. LOC is a density based clustering algorithm that implicitly draws a spatial graph from input domains. Unlike standard clustering algorithms that require specifying the expected number of clusters, we believe that LOC is the first neural network compatible algorithm capable of dynamically choosing the appropriate number of clusters that best fit a given domain. Both ACRBMs and LOC were evaluated in terms of unsupervised learning. Experiments showed that both structures of shallow ACRBMs and AE-based ACRBMs outperformed K-means for image clustering using the same number of clusters. Similarly, LOC outperformed K-means in terms of unsupervised image segmentation.