How does color constancy affect target recognition and instance segmentation?

Shaobing Gao

Abstract: Previous work has demonstrated that incorrect white balance (WB) in the camera image signal processing pipeline has a negative impact on the performance of deep neural networks (DNNs) in high-level vision tasks, and traditional image augmentation approaches are not well suited for modeling WB errors. However, it is still unclear when this impact will occur for which kinds of images and objects. In this paper, we manually labeled 2304 images from the RECommended dataset and NUS dataset and discovered that the effect of WB on DNNs is greatly associated with object size and occlusion level among objects. In images with incorrect WB, small objects and objects with heavily occluded backgrounds are the main factors resulting in the bad performance of DNNs, indicating that the effect of WB is clearly associated with the shape of objects. Our findings may support that the functional role of some neurons in the visual cortex (e.g., V1 or V4 areas) realizing color constancy (CC) and encoding object attributes such as color and shape dependently is to contribute to high-level vision. Furthermore, based on this scientific finding, we proposed a novel augmentation strategy to address the negative impact of incorrect WB by expanding the training datasets in both color transformation and synthetic occlusion. We compared our proposed strategy with the current augmentation strategies and showed that our approach clearly improves the performance of DNNs in detection and segmentation tasks with small objects and objects with heavily occluded backgrounds.