ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation
Abstract: Image-to-video (I2V) generation aims to use the initial frame (alongside a text prompt) to create a video sequence. A grand challenge in I2V generation is to maintain visual consistency throughout the video: existing methods often struggle to preserve the integrity of the subject, background, and style from the first frame, as well as ensure a fluid and logical progression within the video narrative. To mitigate these issues, we propose ConsistI2V, a diffusion-based method to enhance visual consistency for I2V generation. Specifically, we introduce (1) spatiotemporal attention over the first frame to maintain spatial and motion consistency, (2) noise initialization from the low-frequency band of the first frame to enhance layout consistency. These two approaches enable ConsistI2V to generate highly consistent videos. We also extend the proposed approaches to show their potential to improve consistency in auto-regressive long video generation and camera motion control. To verify the effectiveness of our method, we propose I2V-Bench, a comprehensive evaluation benchmark for I2V generation. Our automatic and human evaluation results demonstrate the superiority of ConsistI2V over existing methods.
Submission Length: Regular submission (no more than 12 pages of main content)
Changes Since Last Submission: Change log since last submission:
1. Section 3.4: add discussion on window size K
2. Section 3.5: add discussion with Blurring Diffusion Model
3. Table 1 & Table 2: change text color and highlight
4. Figure 5: change example with less motion
5. Figure 6: add two more ablation study example
6. Table 3: include inference runtime statistics
7. Section 5.4: add discussion on FrameInit hyperparameters; add discussion on runtime efficiency
8. Figure 7: add a new figure for FrameInit hyperparameter comparison
9. Section 5.5 & Figure 8: add discussion & visualization for perspective view change
10. Section 6: move the Limitation section to main paper
11. Appendix C: add more descriptions for I2V-Bench metrics
Code: https://github.com/TIGER-AI-Lab/ConsistI2V
Supplementary Material: zip
Assigned Action Editor: ~Nicolas_THOME2
Submission Number: 2438
Loading