Walking the Tightrope: An Investigation of the Convolutional Autoencoder Bottleneck

Ilja Manakov, Markus Rohm, Volker Tresp

Keywords: autoencoders, unsupervised learning, representation learning, investigation

TL;DR: We investigate the effect of feature map size vs. number of channels in the bottleneck of convolutional autoecoders and find that tuning the former is significantly more important than the latter.

Abstract:

In this paper, we present an in-depth investigation of the convolutional autoencoder (CAE) bottleneck.
Autoencoders (AE), and especially their convolutional variants, play a vital role in the current deep learning toolbox.
Researchers and practitioners employ CAEs for various tasks, ranging from outlier detection and compression to transfer and representation learning.
Despite their widespread adoption, we have limited insight into how the bottleneck shape impacts the CAE's emergent properties.
We demonstrate that increased bottleneck area (i.e., height $\times$ width) drastically improves generalization in terms of reconstruction error while also speeding up training.
The number of channels in the bottleneck, on the other hand, is of secondary importance.
Furthermore, we show empirically that CAEs do not learn to copy their input, even when all layers have the same number of neurons as there are pixels in the input (i.e. there is no bottleneck).
Besides raising important questions for further research, our findings are directly applicable to two of the most common use-cases for CAEs:
In image compression, it is advantageous to increase the feature map size in the bottleneck as this greatly improves reconstruction quality.
For reconstruction-based outlier detection, we recommend decreasing the feature map size so that out-of-distribution samples will yield a higher reconstruction error.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Unsupervised and Self-supervised learning

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:1911.07460/code)