[Re] VAE Approximation Error: ELBO and Exponential Families

Volodymyr Kyrylov, Navdeep Singh Bedi, Qianbo Zang

Keywords: Python, VAE, approximation errors, ELBO, exponential families, deep learning, pytorch, machine learning, rescience c

TL;DR: We reproduce VAE experiments in which increase in ELBO corresponds to decrease in reconstruction quality.

Abstract:

Scope of Reproducibility — Exponential family variational autoencoders struggle with reconstruction when encoders output limited information. We reproduce two experiments in which we first train the decoder and encoder separately. Then, we train both modules jointly using ELBO and observe the degradation of reconstruction. We verify how the theoretical insight into the design of the approximate posterior and decoder distributions for a discrete VAE in a semantic hashing application influences the choice of input features to improve overall performance.

Methodology — We implement and train a synthetic experiment from scratch on a laptop. We use a mix of authors’ code and publicly available code to reproduce a GAN reinterpreted as a VAE. We consult authors’ code to reproduce semantic hashing experiment and make our own implementation. We train models the USI HPC cluster on machines with GTX 1080 Ti or A100 GPUs and 128 GiB of RAM. We spend under 100 GPU hours for all experiments.

Results — We observe expected qualitative behavior predicted by the theory on all experiments. We improve the best semantic hashing model’s test performance by 5 nats by using a modern method for gradient estimation of discrete random variables.

What was easy — Following experiment recipes was easy once we worked out the theory.

What was difficult — The paper enables verification of exponential family distributions VAE designs of arbitrary complexity, which require probabilistic modeling skills. We contribute elaboration on the implementation details of the synthetic experiment and provide code.

Communication with original authors — We are extremely grateful to the authors for discussing the paper, encouraging us to implement experiments on our own, and suggesting directions for improving results over e‐mail.

Paper Url: https://openreview.net/forum?id=OIs3SxU5Ynl

Paper Venue: ICLR 2022

Confirmation: The report pdf is generated from the provided camera ready Google Colab script, The report metadata is verified from the camera ready Google Colab script, The report contains correct author information., The report contains link to code and SWH metadata., The report follows the ReScience latex style guides as in the Reproducibility Report Template (https://paperswithcode.com/rc2022/registration)., The report contains the Reproducibility Summary in the first page., The latex .zip file is verified from the camera ready Google Colab script

Paper Review Url: https://openreview.net/forum?id=OIs3SxU5Ynl

Journal: ReScience Volume 9 Issue 2 Article 37

Doi: https://www.doi.org/10.5281/zenodo.8173745

Code: https://archive.softwareheritage.org/swh:1:dir:6328abd80f26a66924bea7e6b656ebfee9161f31