VideoPanda: Video Panoramic Diffusion With Multi-view Attention

ICLR 2025 Conference Submission7653 Authors

Keywords: video generation, diffusion model, panorama

TL;DR: We enable text and video conditional generation of panorama videos using multiview video models.

Abstract:

High resolution panoramic video content is paramount for immersive experiences in Virtual Reality, but is non-trivial to collect as it requires specialized equipment and intricate camera setups.
In this work, we introduce \ourmodel, a novel approach for synthesizing $360^\circ$ videos conditioned on text or single-view video data. \ourmodel leverages multi-view attention layers to augment a video diffusion model, enabling it to generate consistent multi-view videos that can be combined into immersive panoramic content. \ourmodel is trained jointly using two conditions: text-only and single-view video, and supports autoregressive generation of long-videos.
To overcome the computational burden of multi-view video generation, we randomly subsample the duration and camera views used during training and show that the model is able to gracefully generalize to generating more frames during inference.
Extensive evaluations on both real-world and synthetic video datasets demonstrate that \ourmodel generates more realistic and coherent $360^\circ$ panoramas across all input conditions compared to existing methods.
Visit the project website at https://mvpanovideo.github.io/VideoPanda/ for results.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7653