Improved Image Generation via Sparsity

Roy Ganz, Michael Elad

Keywords: Sparse Coding, Image Generation

TL;DR: Improving image generators by interpreting and regularizing them as manifestations of Convolutional Sparse Coding models.

Abstract:

The interest of the deep learning community in image synthesis has grown massively in recent years. Nowadays, deep generative methods, and specifically Generative Adversarial Networks (GANs),
are leading to state-of-the-art performance, capable of synthesizing images that appear realistic.
While the efforts for improving the quality of the generated images are extensive, most attempts still consider the generator part as an uncorroborated "black-box".
In this paper, we aim to provide a better understanding of the image generation process. We interpret existing generators as implicitly relying on sparsity-inspired models.
More specifically, we show that generators can be viewed as manifestations of the Convolutional Sparse Coding (CSC) and its Multi-Layered version (ML-CSC) synthesis processes.
We leverage this observation by explicitly enforcing a sparsifying regularization on appropriately chosen activation layers in the generator and demonstrate that this leads to improved image synthesis. Furthermore, we show that the same rationale and benefits apply to generators serving inverse problems, demonstrated on the Deep Image Prior (DIP) method.