Depth-Recurrent Residual Connections for Super-Resolution of Real-Time Renderings

Erik Franz, Mengyu Chu, Rüdiger Westermann, Nils Thuerey

TL;DR: A method for persistent latent states in ResBlocks demonstrated for super-resolution of alised image sequences.

Abstract: Inferring temporally coherent data features is crucial for a large variety of learning tasks. We propose a network architecture that introduces temporal recurrent connections for the internal state of the widely used residual blocks. We demonstrate that, with these connections, convolutional neural networks can more robustly learn stable temporal states that persist between evaluations. We demonstrate their potential for inferring high-quality super-resolution images from low resolution images produced with real-time renderers. This data arises in a wide range of applications, and is particularly challenging as it contains a strongly aliased signal. Hence, the data differs substantially from the smooth inputs encountered in natural videos, and existing techniques do not succeed at producing acceptable image quality. We additionally propose a series of careful adjustments of typical generative adversarial architectures for video super-resolution to arrive at a first model that can produce detailed, yet temporally coherent images from an aliased stream of inputs from a real-time renderer.

Keywords: temporal coherence, anti-aliasing, super-resolution, GAN, RNN, real-time rendering