HighlightRemover: Spatially Valid Pixel Learning for Image Specular Highlight Removal

Ling Zhang, Yidong Ma, Zhi Jiang, Weilei He, Zhongyun Bao, Gang Fu, Wenju Xu, Chunxia Xiao

Abstract: Recently, learning-based methods have made significant progress for image specular highlight removal. However, many of these approaches treat all the image pixels as spatially consistent, overlooking the negative impact of invalid pixels on feature reconstruction. This oversight often leads to undesirable outcomes, such as color distortion or residual highlights. In this paper, we propose a novel image specular highlight removal network called HighlightRNet, which utilizes valid pixels as references to reconstruct the highlight-free image. To achieve this, we introduce a context-aware fusion block (CFBlock) that aggregates information in four directions, effectively capturing global contextual information. Additionally, we introduce a location-aware feature transformation module (LFTModule) to adaptively learn the valid pixels for feature reconstruction, thereby avoiding information errors caused by invalid pixels. With these modules, our method can produce high-quality highlight-free results without color distortion and highlight residual. Furthermore, we develop a multiple light image-capturing system to construct a large-scale highlight dataset called NSH, which exhibits minimal misalignment in image pairs and minimal brightness variation in non-highlight regions. Experimental results on various datasets demonstrate the superiority of our method over state-of-the-art methods, both qualitatively and quantitatively.

Primary Subject Area: [Experience] Multimedia Applications

Secondary Subject Area: [Experience] Multimedia Applications

Relevance To Conference: Image specular highlight removal aims to remove specular highlights (light spots) in the image and restore color and texture information in the highlighted regions, which is an image restoration work. This task can improve the effectiveness of some multimedia tasks and visual tasks, such as object detection, semantic segmentation, object tracking, image segmentation. ACM MM conference is highly relevant to the image restoration tasks, including image specular highlight process. At ACMMM conferences, researchers often submit their works on image specular highlight process. These researches not only drive the advancement of the field of image composition but also provide crucial support and guidance for the progress of multimedia applications and technologies.

Submission Number: 3959