ControlVideo: Training-free Controllable Text-to-Video Generation
Outline
Video visualizations
ControlVideo on depth maps
|
|
|
|
|
|
"A charming flamingo gracefully wanders in the calm and serene water, its delicate neck curving into an elegant shape." |
"A striking mallard floats effortlessly on the sparkling pond." |
"A gigantic yellow jeep slowly turns on a wide, smooth road in the city." |
|
|
|
|
|
|
"A sleek boat glides effortlessly through the shimmering river, van gogh style." |
"A majestic sailing boat cruises along the vast, azure sea." |
"A contented cow ambles across the dewy, verdant pasture." |
ControlVideo on canny edges
|
|
|
|
|
|
"A young man riding a sleek, black motorbike through the winding mountain roads." |
"A white swan moving on the lake, cartoon style." |
"A dusty old jeep was making its way down the winding forest road, creaking and groaning with each bump and turn." |
|
|
|
|
|
|
"A shiny red jeep smoothly turns on a narrow, winding road in the mountains." |
"A majestic camel gracefully strides across the scorching desert sands." |
"A fit man is leisurely hiking through a lush and verdant forest." |
ControlVideo on human poses
|
|
|
|
|
|
"James bond moonwalk on the beach, animation style." |
"Hulk is jumping on the street, cartoon style." |
"Goku in a mountain range, surreal style." |
|
|
|
|
|
|
"A man, wearing pink clothes, moonwalk at sunset." |
"The Simpsons in the city, Hockney style." |
"Wonder Woman in a desert, Pop Art style." |
Long video generation
|
|
|
|
"A steamship on the ocean, at sunset, sketch style." |
"Hulk is dancing on the beach, cartoon style." |
Limitations
|
|
|
Source Video |
Structure Sequence |
"Iron man runs in the road." |
Qualitative comparisons
Depth map
Text Prompt: A daring man is scaling a treacherous and jagged peak in the alpine wilderness.
|
|
|
|
|
Source Video |
Structure Sequence |
Tune-A-Video |
Text2Video-Zero |
ControlVideo (Ours) |
Text Prompt: A daring man performing gravity-defying stunts on a high-speed, blue motorbike in an empty parking lot.
|
|
|
|
|
Source Video |
Structure Sequence |
Tune-A-Video |
Text2Video-Zero |
ControlVideo (Ours) |
Text Prompt: A dusty old jeep was making its way down the winding forest road, creaking and groaning with each bump and turn.
|
|
|
|
|
Source Video |
Structure Sequence |
Tune-A-Video |
Text2Video-Zero |
ControlVideo (Ours) |
Canny edge
Text Prompt: A curious golden dog curiously wanders on the rocky mountain trail.
|
|
|
|
|
Source Video |
Structure Sequence |
Tune-A-Video |
Text2Video-Zero |
ControlVideo (Ours) |
Text Prompt: A mighty elephant marches steadily through the rugged terrain.
|
|
|
|
|
Source Video |
Structure Sequence |
Tune-A-Video |
Text2Video-Zero |
ControlVideo (Ours) |
Text Prompt: A shiny silver vehicle gracefully maneuvers towards a modern glass building.
|
|
|
|
|
Source Video |
Structure Sequence |
Tune-A-Video |
Text2Video-Zero |
ControlVideo (Ours) |
Human Pose
Text Prompt: Iron man does the moonwalk in the road.
|
|
|
|
|
|
Source Video |
Structure Sequence |
Tune-A-Video |
Text2Video-Zero |
Follow-Your-Pose |
ControlVideo (Ours) |
Text Prompt: A robot dances on a road, animation style.
|
|
|
|
|
|
Source Video |
Structure Sequence |
Tune-A-Video |
Text2Video-Zero |
Follow-Your-Pose |
ControlVideo (Ours) |
Text Prompt: The astronaut dances in futuristic city, cyberpunk style.
|
|
|
|
|
|
Source Video |
Structure Sequence |
Tune-A-Video |
Text2Video-Zero |
Follow-Your-Pose |
ControlVideo (Ours) |
Ablation studies
Effect of fully cross-frame interaction and interleaved-frame smoother
Text Prompt: A mighty elephant marches steadily through the rugged terrain.
|
|
|
|
|
|
Source video |
Individual |
First-only |
Sparse-Causal |
Fully Cross-frame |
Fully + Smoother |
Text Prompt: A dusty old jeep was making its way down the winding forest road, creaking and groaning with each bump and turn.
|
|
|
|
|
Structure Sequence |
w/o smoother |
Timesteps {0,1} |
Timesteps {30,31} |
Timesteps {48,49} |
How many timesteps are used in interleaved-frame smoother?
Text Prompt: A sleek black jeep was speeding along the narrow forest road, dodging trees and rocks.
|
|
|
|
|
|
Structure Sequence |
0 step |
2 steps |
4 steps |
6 steps |
8 steps |