Color4E: Event Demosaicing for Full-color Event Guided Image Deblurring

Yi Ma, Peiqi Duan, Yuchen Hong, Chu Zhou, Yu Zhang, Jimmy Ren, Boxin Shi

Abstract: Neuromorphic event sensors are novel visual cameras that feature high-speed illumination-variation sensing and have found widespread application in guiding frame-based imaging enhancement. This paper focuses on color restoration in the event-guided image deblurring task, we fuse blurry images with mosaic color events instead of mono events to avoid artifacts such as color bleeding. The challenges associated with this approach include demosaicing color events for reconstructing full-resolution sampled signals and fusing bimodal signals to achieve image deblurring. To meet these challenges, we propose a novel network called Color4E to enhance the color restoration quality for the image deblurring task. Color4E leverages an event demosaicing module to upsample the spatial resolution of mosaic color events and a cross-encoding image deblurring module for fusing bimodal signals, a refinement module is designed to fuse full-color events and refine initial deblurred images. Furthermore, to avoid the real-simulated gap of events, we implement a display-filter-camera system that enables mosaic and full-color event data captured synchronously, to collect a real-captured dataset used for network training and validation. The results on the public dataset and our collected dataset show that Color4E enables high-quality event-based image deblurring compared to state-of-the-art methods.

Primary Subject Area: [Content] Multimodal Fusion

Secondary Subject Area: [Experience] Multimedia Applications

Relevance To Conference: This paper explores the combination and fusion of two modalities, i.e., image signals and color event signals, to enhance the performance of image deblurring. The proposed method proposes a fusion network for events and images, in which data forms are different from each other, to realize the complementarity of the two modalities, which may help researchers in processing and feature fusion of similar multimodal signals.

Submission Number: 2324