JoReS-Diff: Joint Retinex and Semantic Priors in Diffusion Model for Low-light Image Enhancement

Yuhui Wu, Guoqing Wang, Zhiwen Wang, Yang Yang, Tianyu Li, Malu Zhang, Chongyi Li, Heng Tao Shen

Abstract: Low-light image enhancement (LLIE) has achieved promising performance by employing conditional diffusion models. Despite the success of some conditional methods, previous methods may neglect the importance of a sufficient formulation of task-specific condition strategy, resulting in suboptimal visual outcomes. In this study, we propose JoReS-Diff, a novel approach that incorporates Retinex- and semantic-based priors as the additional pre-processing condition to regulate the generating capabilities of the diffusion model. We first leverage pre-trained decomposition network to generate the Retinex prior, which is updated with better quality by an adjustment network and integrated into a refinement network to implement Retinex-based conditional generation at both feature- and image-levels. Moreover, the semantic prior is extracted from the input image with an off-the-shelf semantic segmentation model and incorporated through semantic attention layers. By treating Retinex- and semantic-based priors as the condition, JoReS-Diff presents a unique perspective for establishing an diffusion model for LLIE and similar image enhancement tasks. Extensive experiments validate the rationality and superiority of our approach.

Relevance To Conference: This work explores an insightful condition strategy by introducing multimodal priors, which shows the superior capability in resolving the diffusion-based image restoration and enhancement tasks. The interaction and cooperation of the multimodal priors provide the sufficient guidance to control the diffusion model and handle well with various low-level vision tasks. And the idea of integrating multimodal priors can be extended in the future through two aspects, i.e. the extension of more kinds of multimodal priors and the generalization on other low-level vision tasks.

Primary Subject Area: [Content] Media Interpretation

Submission Number: 707