HERO: Harnessing Temporal Modeling for Diffusion-Based Video Outpainting
Keywords: video outpainting, diffusion model
TL;DR: Enhancing diffusion-based video outpainting from a temporal modeling perspective
Abstract: Video outpainting expands the spatial perspective of a video, enabling it to adapt to various display devices with different aspect ratios.
Current diffusion-based approaches for video outpainting often suffer from quality issues such as blurred details, local distortion, and temporal instability, significantly impacting the user experience.
The root cause is the insufficient temporal modeling in video
outpainting, which inadequately represents the relationships between frames over time.
To address this issue, a novel approach called HERO~(Harnessing the tEmpoRal modeling for diffusion-based Outpainting) is proposed to effectively tackles these generated video quality problems.
HERO employs two critical components to enhance temporal modeling: the Temporal Reference Module, which provides reference features that extend beyond spatial dimensions; and the Interpolation-based Motion Modelling Module, designed to stabilize generated frames.
By integrating these modules, these quality issues in video outpainting are effectively addressed.
Extensive experiments on multiple benchmarks demonstrate that HERO outperforms existing methods qualitatively and quantitatively.
Supplementary Material: zip
Primary Area: generative models
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 628
Loading