# LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models

Paper ID: 2758

## Text-to-Video Generation

## (Click image to play video)

-->
Cinematic shot of Van Gogh's selfie, Van Gogh style
A corgi’s head depicted as an explosion of a nebula, high quality
A panda drinking coffee in a cafe in Paris
Iron Man flying in the sky
A jellyfish floating through the ocean, with bioluminescent tentacles
A Mars rover moving on Mars
The bund Shanghai, oil painting
A fantasy landscape, trending on artstation, 4k, high resolution
A space shuttle launching into orbit, with flames and smoke billowing out from the engines
A super cool giant robot in Cyberpunk city, artstation
A tropical beach at sunrise, with palm trees and crystal-clear water in the foreground
A boat sailing leisurely along the Seine River with the Eiffel Tower in background by Vincent van Gogh
Gwen Stacy reading a book
A future where humans have achieved teleportation technology
A steam train moving on a mountainside by Vincent van Gogh
Yoda playing guitar on the stage
A beautiful coastal beach in spring, waves lapping on sand by Hokusai, in the style of Ukiyo
A happy fuzzy panda playing guitar nearby a campfire, snow mountain in the background
A cat eating food out of a bowl, in style of Van Gogh
A boat sailing leisurely along the Seine River with the Eiffel Tower in background by Vincent van Gogh
Vincent van Gogh is painting in the room

## Long Video Generation

## (Click image to play video)

-->
A panda playing guitar near a campfire, snow mountain in the background.
A car moving on an empty street, rainy evening, Van Gogh painting.
Albert Einstein is reading a paper.

## Personalized T2V Generation

## (Click image to play video)

Training image samples

Misaka Mikoto walking in the city.
Misaka Mikoto walking in the space.
Misaka Mikoto walking in the space.
Misaka Mikoto.

## Vimeo25M samples

## (Click image to play video)

A bride and groom walk down the aisle of a church with people in.
A man standing at a podium giving a speech.
A sunset with clouds in the sky.
An aerial view of a large estate.

## Comparison (Joint image-video fine-tuning)

## (Click image to play video)

Iron Man dancing on the beach (joint image-video fine-tuning)
A teddy bear playing in the water (joint image-video fine-tuning)
Iron Man dancing on the beach (only video fine-tuning)
A teddy bear playing in the water (only video fine-tuning)
Iron Man dancing on the beach (only video fine-tuning)
A teddy bear playing in the water (only video fine-tuning)
Iron Man dancing on the beach (only video fine-tuning)
A teddy bear playing in the water (only video fine-tuning)

## Comparison (RoPE)

## (Click image to play video)

A teddy bear walking on the street. (16 frames)
A panda taking selfie. (16 frames)
A teddy bear walking on the street. (32 frames)
A panda taking selfie. (32 frames)
A teddy bear walking on the street. (48 frames)
A panda taking selfie. (48 frames)
A teddy bear walking on the street. (64 frames)
A panda taking selfie. (64 frames)

## Abstract

We present LaVie, an innovative system for text-to-video (T2V) generation that operates on the foundation of a cascade of video latent diffusion models. Comprising three networks, namely a base T2V model, a temporal interpolation model, and a video super-resolution model, LaVie aims to address research question of extending a pre-trained text-to-image (T2I) model into realm of video synthesis. Our objective is to accomplish the synthesis of visually realistic and temporally coherent videos while preserving the strong compositional nature of the model. We found that the incorporation of simple temporal self-attentions, coupled with relative positional encoding, adequately captures the temporal correlations inherent in video data. Additionally, we validate that the process of joint image-video fine-tuning plays a pivotal role in producing high-quality outcomes. To enhance the performance of LaVie, we curate a comprehensive and diverse video dataset named Vimeo25M, consisting of 25 million text-video pairs that prioritize quality, diversity, and aesthetic appeal. Experimental evaluations demonstrate that LaVie outperforms state-of-the-art in terms of quantitative and qualitative assessments. Furthermore, we showcase the versatility of pre-trained LaVie models in long video generation and personalized video synthesis applications.