Generalizing ISP Model by Unsupervised Raw-to-raw Mapping

Dongyu Xie, Chaofan Qiao, Lanyue Liang, Zhiwen Wang, Tianyu Li, Qiao Liu, Chongyi Li, Guoqing Wang, Yang Yang

Abstract: ISP (Image Signal Processor) serves as a pipeline converting unprocessed raw images to sRGB images, positioned before nearly all visual tasks. Due to the varying spectral sensitivities of cameras, raw images captured by different cameras exist in different color spaces, making it challenging to deploy ISP across cameras with consistent performance. To address this challenge, it is intuitively to incorporate a raw-to-raw mapping (mapping raw images across camera color spaces) module into the ISP. However, the lack of paired data (i.e., images of the same scene captured by different cameras) makes it difficult to train a raw-to-raw model using supervised learning methods. In this paper, we aim to achieve ISP generalization by proposing the first unsupervised raw-to-raw model. To be specific, we propose a CSTPP (Color Space Transformation Parameters Predictor) module to predict the space transformation parameters in a patch-wise manner, which can accurately perform color space transformation and flexibly manage complex lighting conditions. Additionally, we design a CycleGAN-style training framework to realize unsupervised learning, overcoming the deficiency of paired data. Our proposed unsupervised model achieved performance comparable to that of the state-of-the-art semi-supervised method in raw-to-raw task. Furthermore, to assess its ability to generalize the ISP model across different cameras, we for the first formulated cross-camera ISP task and demonstrated the performance of our method through extensive experiments. Codes will be publicly available.

Primary Subject Area: [Content] Media Interpretation

Secondary Subject Area: [Generation] Generative Multimedia, [Systems] Systems and Middleware

Relevance To Conference: The Image Signal Processor (ISP) transforms the real world captured by the cameras into digital images or videos. The majority of visual media information requiring processing by the ISP system. This paper for the first time discusses the issue of generalization of the entire ISP system under different cameras and proposes a method to enhance the generalization of the ISP system, enabling the ISP to be deployed under different cameras. We believe that our work will contribute to enhancing the quality and increasing the quantity of visual media information.

Submission Number: 5425