Importance weighted compression

Lucas Theis, Jonathan Ho

Keywords: compression, variational autoencoder, importance weighted autoencoder, bits-back coding, softmin coding, gumbel

TL;DR: The loss of an IWAE can be interpreted as the coding cost of a lossless compression scheme or the rate-distortion trade-off of a lossy compression scheme

Abstract: The connection between variational autoencoders (VAEs) and compression is well established and they have been used for both lossless and lossy compression. Compared to VAEs, importance-weighted autoencoders (IWAEs) achieve a larger bound on the log-likelihood. However, it is not well understood whether a similar connection between IWAEs and compression exists and whether the improved loss corresponds to better compression performance. Here we show that the loss of IWAEs can indeed be interpreted as the cost of lossless or lossy compression schemes, and using IWAEs for compression can lead to small improvements in performance.