LumiSculpt: A Consistency Lighting Control Network for Video Generation
Keywords: Light Control, Video generation, Text-to-video generation, diffusion models
TL;DR: A new method to achieve lighting control in text-to-video generation.
Abstract: Lighting is essential for the naturalness of video generation, which significantly impacts the overall aesthetic quality of the generated video. However, due to the deep coupling between lighting and the temporal features of videos, it is challenging for modeling independent and coherent lighting attributes, resulting in a lack of approaches for controlling lighting in videos. Therefore, inspired by the established controllable T2I models, we propose LumiSculpt, achieving precise and consistency lighting control in video generation models for the first time. LumiSculpt equips the video generation with strong interactive capabilities, allowing for the input of custom lighting reference image sequences. Furthermore, the core learnable plug-and-play module of LumiSculpt enables us to achieve remarkable performance on controlling light intensity, position, trajectory in latent video diffusion models based on the advanced DiT backbone. Additionally, to effectively fine-tune LumiSculpt and address the issue of insufficient lighting data, we construct LumiHuman, a new lightweight and flexible dataset for portrait lighting of images and videos. Experiments demonstrate that LumiSculpt achieves precise and high-quality lighting control in video generation. The code, model, and dataset will be released to facilitate further research. Video results are shown in the supplementary material.
Supplementary Material: zip
Primary Area: generative models
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 4419
Loading