ControlVideo: Training-free Controllable Text-to-Video Generation

Outline

Video visualizations

ControlVideo on depth maps

"A charming flamingo gracefully wanders in the calm and serene water, its delicate neck curving into an elegant shape." "A striking mallard floats effortlessly on the sparkling pond." "A gigantic yellow jeep slowly turns on a wide, smooth road in the city."
"A sleek boat glides effortlessly through the shimmering river, van gogh style." "A majestic sailing boat cruises along the vast, azure sea." "A contented cow ambles across the dewy, verdant pasture."

ControlVideo on canny edges

"A young man riding a sleek, black motorbike through the winding mountain roads." "A white swan moving on the lake, cartoon style." "A dusty old jeep was making its way down the winding forest road, creaking and groaning with each bump and turn."
"A shiny red jeep smoothly turns on a narrow, winding road in the mountains." "A majestic camel gracefully strides across the scorching desert sands." "A fit man is leisurely hiking through a lush and verdant forest."

ControlVideo on human poses

"James bond moonwalk on the beach, animation style." "Hulk is jumping on the street, cartoon style." "Goku in a mountain range, surreal style."
"A man, wearing pink clothes, moonwalk at sunset." "The Simpsons in the city, Hockney style." "Wonder Woman in a desert, Pop Art style."

Long video generation

"A steamship on the ocean, at sunset, sketch style." "Hulk is dancing on the beach, cartoon style."

Limitations

Source Video Structure Sequence "Iron man runs in the road."

Qualitative comparisons

Depth map

Text Prompt: A daring man is scaling a treacherous and jagged peak in the alpine wilderness.

Source Video Structure Sequence Tune-A-Video Text2Video-Zero ControlVideo (Ours)

Text Prompt: A daring man performing gravity-defying stunts on a high-speed, blue motorbike in an empty parking lot.

Source Video Structure Sequence Tune-A-Video Text2Video-Zero ControlVideo (Ours)

Text Prompt: A dusty old jeep was making its way down the winding forest road, creaking and groaning with each bump and turn.

Source Video Structure Sequence Tune-A-Video Text2Video-Zero ControlVideo (Ours)

Canny edge

Text Prompt: A curious golden dog curiously wanders on the rocky mountain trail.

Source Video Structure Sequence Tune-A-Video Text2Video-Zero ControlVideo (Ours)

Text Prompt: A mighty elephant marches steadily through the rugged terrain.

Source Video Structure Sequence Tune-A-Video Text2Video-Zero ControlVideo (Ours)

Text Prompt: A shiny silver vehicle gracefully maneuvers towards a modern glass building.

Source Video Structure Sequence Tune-A-Video Text2Video-Zero ControlVideo (Ours)

Human Pose

Text Prompt: Iron man does the moonwalk in the road.

Source Video Structure Sequence Tune-A-Video Text2Video-Zero Follow-Your-Pose ControlVideo (Ours)

Text Prompt: A robot dances on a road, animation style.

Source Video Structure Sequence Tune-A-Video Text2Video-Zero Follow-Your-Pose ControlVideo (Ours)

Text Prompt: The astronaut dances in futuristic city, cyberpunk style.

Source Video Structure Sequence Tune-A-Video Text2Video-Zero Follow-Your-Pose ControlVideo (Ours)

Ablation studies

Effect of fully cross-frame interaction and interleaved-frame smoother

Text Prompt: A mighty elephant marches steadily through the rugged terrain.

Source video Individual First-only Sparse-Causal Fully Cross-frame Fully + Smoother

Which timesteps does interleaved-frame smoother perform?

Text Prompt: A dusty old jeep was making its way down the winding forest road, creaking and groaning with each bump and turn.

Structure Sequence w/o smoother Timesteps {0,1} Timesteps {30,31} Timesteps {48,49}

How many timesteps are used in interleaved-frame smoother?

Text Prompt: A sleek black jeep was speeding along the narrow forest road, dodging trees and rocks.

Structure Sequence 0 step 2 steps 4 steps 6 steps 8 steps