EdgeRelight360: Text-Conditioned 360-Degree HDR Image Generation for Real-Time On-Device Video Portrait Relighting

Michel Sarkis, Mahesh Reddy, Guillaume Berger, Min-Hui Lin, Fatih Porikli, Ning Bi

Abstract:

In this paper, we present EdgeRelight360, an approach
for real-time video portrait relighting on mobile devices,
utilizing text-conditioned generation of 360-degree high dynamic range image (HDRI) maps. Our method proposes a
diffusion-based text-to-360-degree image generation in the
HDR domain, taking advantage of the HDR10 standard.
This technique facilitates the generation of high-quality, realistic lighting conditions from textual descriptions, offering
flexibility and control in portrait video relighting task. Unlike the previous relighting frameworks, our proposed system performs video relighting directly on-device, enabling
real-time inference with real 360-degree HDRI maps. This
on-device processing ensures both privacy and guarantees
low runtime, providing an immediate response to changes in
lighting conditions or user inputs. Our approach paves the
way for new possibilities in real-time video applications, including video conferencing, gaming, and augmented reality,
by allowing dynamic, text-based control of lighting conditions.