Direct Evolutionary Optimization of Variational Autoencoders With Binary Latents

Enrico Guiraud, Jakob Drefs, Filippos S Panagiotou, Jorg Lucke

Keywords: variational optimization, variational autoencoders, denoising, inpainting, evolutionary algorithms

Abstract:

Many types of data are generated at least partly by discrete causes that are sparsely
active. To model such data, we here investigate a deep generative model in the
form of a variational autoencoder (VAE) which can learn a sparse, binary code
for its latents. Because of the latents’ discrete nature, standard VAE training is
not possible. The goal of previous approaches has therefore been to amend (i.e.,
typically anneal) discrete priors in order to train discrete VAEs analogously to
conventional ones. Here, we divert much more strongly from conventional VAE
training: We ask if it is also possible to keep the discrete nature of the latents
fully intact by applying a direct, discrete optimization for the encoding model. In
doing so, we (1) sidestep standard VAE mechanisms such as sampling approximation, reparameterization trick and amortization, and (2) observe a much sparser
encoding compared to autoencoders that use annealed discrete latents. Direct optimization of VAEs is enabled by an evolutionary algorithm in conjunction with
truncated posteriors as variational distributions, i.e. by a combination of methods
which is here for the first time applied to a deep model. We first show how the discrete variational method (A) ties into gradient ascent for network weights, and how
it (B) uses the decoder network to select binary latent states for training. Sparse
codes have prominently been applied to image patches, where latents encode edge-like structure. For our VAEs, we maintain this prototypical application domain
and observe the emergence of much sparser codes compared to more conventional
VAEs. To allow for a broad comparison to other approaches, the emerging encoding was then evaluated on denoising and inpainting tasks, which are canonically
benchmarks for image patch models. For datasets with many, large images of single objects (ImageNet, CIFAR etc) deep generative models with dense codes seem
preferable. For image patches, however, we observed advantages of sparse codes
that give rise to state-of-the-art performance in ‘zero-shot’ denoising and inpainting benchmarks. Sparse codes can consequently make VAEs competitive on tasks
where they have previously been outperformed by non-generative approaches.