Unsupervised Video-to-Video Translation via Self-Supervised Learning

Kangning Liu, Shuhang Gu, Radu Timofte

TL;DR: A temporally consistent and modality flexible unsupervised video-to-video translation framework trained in a self-supervised manner.

Abstract: Existing unsupervised video-to-video translation methods fail to produce translated videos which are frame-wise realistic, semantic information preserving and video-level consistent. In this work, we propose a novel unsupervised video-to-video translation model. Our model decomposes the style and the content, uses specialized encoder-decoder structure and propagates the inter-frame information through bidirectional recurrent neural network (RNN) units. The style-content decomposition mechanism enables us to achieve long-term style-consistent video translation results as well as provides us with a good interface for modality flexible translation. In addition, by changing the input frames and style codes incorporated in our translation, we propose a video interpolation loss, which captures temporal information within the sequence to train our building blocks in a self-supervised manner. Our model can produce photo-realistic, spatio-temporal consistent translated videos in a multimodal way. Subjective and objective experimental results validate the superiority of our model over the existing methods.

Keywords: Self-supervised learning, Unsupervised video-to-video translation