Pivotal Prompt Tuning for Video Dynamic Editing

Sunjae Yoon, GwanHyeong Koo, Ji Woo Hong, Joshua Tian Jin Tee, Chang D. Yoo

Primary Area: representation learning for computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Video Editing, Multi-modal video generation, Prompt Analysis, Diffusion model

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: This paper proposes video editing framework that performs dynamic motion editing via pivotal prompt tuning.

Abstract: Text-conditioned image editing has recently provided high-quality edits on images based on diffusion frameworks. Unfortunately, this success did not carry over to video editing, which continues to be challenging. Video editing is limited to rigid editing such as object overlay and style transfer. This paper proposes pivotal dynamic editing (PDEdit) for performing spatial-temporal non-rigid video editing based only on the target text, which has never been attempted before. PDEdit is capable of changing the motion of an object/person in the video, either at a specific moment or throughout the video, while preserving the temporal consistency of edited motions and a high level of fidelity to the original input video. In contrast to previous works, the proposed method performs editing based only on the input video and target text. It does not require any other auxiliary inputs (e.g., object masks or source video captions). Based on the video diffusion model, PDEdit using the proposed prompt pivoting leverages the target text prompt for editing the input video. The quality and adaptability of the proposed method on numerous input videos from different domains show the proposed to be highly effective. It can produce high-fidelity video edits under a single unified PDEdit framework. The code for this work will be made publicly available.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6930