INR-V: A Continuous Representation Space for Video-based Generative Tasks

Bipasha Sen, Aditya Agarwal, Vinay P Namboodiri, C.V. Jawahar

Abstract: Generating videos is a complex task that is accomplished by generating a set of temporally coherent images frame-by-frame. This limits the expressivity of videos to only image-based operations on the individual video frames needing network designs to obtain temporally coherent trajectories in the underlying image space. We propose INR-V, a video representation network that learns a continuous space for video-based generative tasks. INR-V parameterizes videos using implicit neural representations (INRs), a multi-layered perceptron that predicts an RGB value for each input pixel location of the video. The INR is predicted using a meta-network which is a hypernetwork trained on neural representations of multiple video instances. Later, the meta-network can be sampled to generate diverse novel videos enabling many downstream video-based generative tasks. Interestingly, we find that conditional regularization and progressive weight initialization play a crucial role in obtaining INR-V. The representation space learned by INR-V is more expressive than an image space showcasing many interesting properties not possible with the existing works. For instance, INR-V can smoothly interpolate intermediate videos between known video instances (such as intermediate identities, expressions, and poses in face videos). It can also in-paint missing portions in videos to recover temporally coherent full videos. In this work, we evaluate the space learned by INR-V on diverse generative tasks such as video interpolation, novel video generation, video inversion, and video inpainting against the existing baselines. INR-V significantly outperforms the baselines on several of these demonstrated tasks, clearly showing the potential of the proposed representation space.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission: Camera Ready Version. All the comments of the reviewers and the AE have been incorporated in this version. Additionally, we have also uploaded the camera-ready supplementary video.

Video: https://skymanaditya1.github.io/INRV/

Code: https://skymanaditya1.github.io/INRV/

Assigned Action Editor: ~Joao_Carreira1

License: Creative Commons Attribution 4.0 International (CC BY 4.0)

Submission Number: 210