ActAnywhere
Subject-Aware Video Background Generation

NeurIPS 2024
SUBMISSION 21087

Videos should play automatically and in a loop. If a video does not play automatically, we show the control bar for users to play.
Despite the various ways one may choose to generate the composited frames, we used Adobe Firefly to generate those for the results shown here. Hover mouse over the composited frames to see their corresponding text prompts, which are either produced from ChatGPT 4 or manually written. Please refer to Sec. 6.2 in the appendix for details of this process.

Video background generation with composited frame conditioning
Original video
(not used as model input)
Segmentation
Woman running on water
Condition
Output
Original video
(not used as model input)
Segmentation
Mallard wandering around a firepit.
Condition
Output
Original video
(not used as model input)
Segmentation
A man folding bed sheets.
Condition
Output
Original video
(not used as model input)
Segmentation
Purple tie-dye jogger runs in serene park, mist over lake.
Condition
Output
Original video
(not used as model input)
Segmentation
A woman is water-skiing.
Condition
Output
Original video
(not used as model input)
Segmentation
A dog plays beside an old man.
Condition
Output
Video background generation with background-only frame conditioning
Original video
(not used as model input)
Segmentation
Condition
Output
Original video
(not used as model input)
Segmentation
Condition
Output
Original video
(not used as model input)
Segmentation
Condition
Output
Diverse generated camera motion
Segmentation
Lost in thought, figure strolls through foggy cityscape in winter attire.
Condition
Seed 1
Seed 2
Seed 3
Seed 4
Segmentation
Infant in blue onesie explores a toy-filled nursery.
Condition
Seed 1
Seed 2
Seed 3
Seed 4
Segmentation
Child in blue jacket joyfully picks a pumpkin in autumn patch.
Condition
Seed 1
Seed 2
Seed 3
Segmentation
Traveler, backpack in tow, seeks secrets in desolate landscape's vastness.
Condition
Seed 1
Seed 2
Seed 3
Segmentation
Immersed gamer moves intensely in high-tech room, exploring virtual reality.
Condition
Seed 1
Seed 2
Seed 3
Different backgrounds with the same foreground
Woman in red faces vast grey, reflecting an inner journey
Original video
Segmentation
Condition 1
Output 1
Condition 2
Output 2
Condition 3
Output 3
Condition 4
Output 4
Condition 5
Output 5
Condition 6
Output 6
Condition 7
Output 7
Woman poised backstage, ready for defining theater spotlight moment.
Original video
Segmentation
Condition 1
Output 1
Condition 2
Output 2
Condition 3
Output 3
Condition 4
Output 4
Determined athlete runs through cool, overcast weather, undeterred in the morning.
Original video
Segmentation
Condition 1
Output 1
Condition 2
Output 2
Condition 3
Output 3
A determined athlete trains in diverse landscapes for marathon endurance.
Original video
Segmentation
Condition 1
Output 1
Condition 2
Output 2
Woman confidently at outdoor, engaging at sunset.
Original video
Segmentation
Condition 1
Output 1
Condition 2
Output 2
Condition 3
Output 3
Condition 4
Output 4
Diverse generated contents
Segmentation
Traveler, backpack in tow, seeks secrets in desolate landscape's vastness.
Condition
Seed 1
Seed 2
Seed 3
Seed 4
Segmentation
A child creating shimmering soap bubbles at a grassland.
Condition
Seed 1
Seed 2
Seed 3
Seed 4
Segmentation
Child in beach attire joyfully runs shore, bucket in hand, playing.
Condition
Seed 1
Seed 2
Condition frame of a different subject
Original video
(not used as model input)
Segmentation
A man is holding a balloon, and floating up by the balloon.
Condition
Output
Original video
(not used as model input)
Segmentation
Cyclist pauses, admires scenic overlook with open road and tranquil landscape.
Condition
Output
Comparison with baselines with original videos

Here we show the video version of Fig. 4 in the main manuscript. Baselines accepting video input see the original videos.

A car drifting on a snowy mountain road
Original video
Segmentation
Condition
Ours
Gen1 [9]
Text2LIVE [3]
TokenFlow [12]
Control-A-Video [7]
AnimateDiff [13]
VideoCrafter1 [6]
A woman performing motorcycle stunts
Original video
Segmentation
Condition
Ours
Gen1 [9]
Text2LIVE [3]
TokenFlow [12]
Control-A-Video [7]
AnimateDiff [13]
VideoCrafter1 [6]