Efficient architectural aspects for text-to-video generation pipeline

Arkhipkin Sergeevich Vladimir, Zein Shaheen, Viacheslav Vasilev, Denis Valerievich Dimitrov, Andrey Kuznetsov

Keywords: text-to-video, video generation, temporal consistency, frames interpolation, Inception Score, CLIPSIM, MoVQ video decoder

TL;DR: In this paper we propose a two-stage latent diffusion video generation architecture and a new MoVQ video decoding scheme. We also conduct experimental research to compare temporal blocks and temporal layers for architecture design.

Abstract:

Multimedia generation approaches occupy a prominent place in artificial intelligence research. Text-to-image models achieved high quality results over the last years, however video synthesis methods recently started to develop. In this paper we present a new two-stage latent diffusion video generation architecture using a new MoVQ video decoding scheme. The first stage concerns keyframes synthesis, while the second one is devoted to interpolated frames generation. We compare two temporal conditioning approaches during evaluation and show the improvement of using temporal blocks over temporal layers in terms of IS and CLIPSIM metrics reflecting video generation quality aspects. We also evaluate different configurations of MoVQ-based video decoding scheme to achieve higher PSNR, SSIM, MSE and LPIPS scores. Finally, we compare our pipeline with existing solutions and achieve top-3 CLIPSIM metric score (0.2976).

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 9160