Video Generation Beyond a Single Clip

Hsin-Ping Huang, Yu-Chuan Su, Ming-Hsuan Yang

Keywords: Video generation

Abstract: This work tackles the challenge of generating long videos, which entails producing videos that surpass the output length of video generation models. Due to computational constraints, video generation models are restricted to generating relatively short video clips compared to the length of real-world videos. Existing approaches employ a sliding window technique to generate long videos during inference, but this method is often restricted to homogeneous content and recurring events. To generate long videos that encompass diverse content and multiple events, we propose utilizing additional guidance to steer the video generation process. We further introduce a multi-stage approach to address this challenge, enabling us to leverage existing video generation models to produce high-quality videos within a limited time window while holistically modeling the long video based on the provided guidance. Our method complements existing video generation efforts. Extensive experiments on challenging real-world videos demonstrate the advantages of the proposed method, which outperforms the state-of-the-art by up to 9.5\% in objective metrics and is preferred by users over 80\% of the time. The source code and trained models will be released to the public.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 466