MoViE: Mobile Diffusion for Video Editing

Adil Karjauv, Noor Fathima, Ioannis Lelekas, Fatih Porikli, Amir Ghodrati, Amir Habibian

Keywords: diffusion, video generation, video editing, efficient, on-device, mobile, video-to-video

TL;DR: MoViE is the first diffusion-based video editing model to run in (near) real time on mobile devices, demonstrating 12 FPS (frames-per-second) editing speed.

Abstract: Recent progress in diffusion-based video editing techniques has shown remarkable potential and is being increasingly utilized in practical applications. However, these methods remain prohibitively expensive and particularly challenging to deploy on mobile devices. In this study, we introduce a series of optimizations that render mobile video editing feasible. Building upon the existing image editing model, we first optimize its architecture and incorporate a lightweight autoencoder. Subsequently, we propose a new classifier-free guidance distillation with multiple modalities, resulting in a $3×$ on-device speed-up. Finally, we reduce the number of sampling steps to one ($10×$ speed-up) by introducing a novel adversarial distillation scheme which preserves the controllability of the editing process in contrast to previous arts. Collectively, these optimizations enable video editing at an impressive $12$ frames per second on mobile devices, while maintaining high editing quality.

Primary Area: generative models

Submission Number: 19536