Beyond a Video Frame Interpolator: A Space Decoupled Learning Approach to Continuous Image Transition

Yang Tao, Peiran Ren, Xuansong Xie, Xian-Sheng Hua, Lei Zhang

Keywords: Video Frame Interpolation, Continuous Image Transition, Image Synthesis, Space Decoupled Learning

Abstract: Video frame interpolation (VFI) aims to improve the temporal resolution of a video sequence. Most of the existing deep learning based VFI methods adopt off-the-shelf optical flow algorithms to estimate the bidirectional flows and interpolate the missing frames accordingly. Though having achieved a great success, these methods require much human experience to tune the bidirectional flows and often generate unpleasant results when the estimated flows are not accurate. In this work, we rethink the VFI problem and formulate it as a continuous image transition (CIT) task, whose key issue is to transition an image from one space to another space continuously. More specifically, we learn to implicitly decouple the images into a translatable flow space and a non-translatable feature space. The former depicts the translatable states between the given images, while the later aims to reconstruct the intermediate features that cannot be directly translated. In this way, we can easily perform image interpolation in the flow space and intermediate image synthesis in the feature space, obtaining a CIT model. The proposed space decoupled learning (SDL) approach is simple to implement, while it provides an effective framework to a variety of CIT problems beyond VFI, such as style transfer and image morphing. Our extensive experiments on a variety of CIT tasks demonstrate the superiority of SDL to existing methods. Codes will be made publicly available.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2203.09771/code)