When Glass Disappears at Night: A Novel NIR-RGB Multimodal Solution

Abstract: Glass surface detection (GSD) has recently been attracting research interests. However, existing GSD methods focus on modeling glass surface properties for daytime scenes only, and can easily fail in nighttime scenes due to significant lighting discrepancies. We observe that, due to the spectral differences between Near-Infrared (NIR) light sources and common LED lights, NIR and RGB cameras capture complementary visual patterns (\eg, light reflections, shadows, and edges) of glass surfaces, and cross-comparing their lighting and reflectance properties can provide reliable cues for nighttime GSD. Inspired by this observation, we propose a novel approach for nighttime GSD based on the multi-modal NIR and RGB image pairs. We first construct a nighttime GSD dataset, which contains $6,192$ RGB-NIR image pairs captured in diverse real-world nighttime scenes, with corresponding carefully-annotated glass surface masks. We then propose a novel network for the nighttime GSD task with two novel modules: (1) an RGB-NIR Guidance Enhancement (RNGE) module for extracting and enriching the NIR reflectance features with the guidance of RGB reflectance features, and (2) an RGB-NIR Fusion and Localization (RNFL) module for fusing RGB and NIR reflectance features into glass features conditioned on the multi-modal illumination discrepancy-aware features. Extensive experiments demonstrate that our method outperforms state-of-the-art methods in nighttime scenes while generalizing well to daytime scenes. We will release our dataset and codes.

Submission Type: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

We thank the AE for handling our paper and the reviewers for their constructive review comments. We have addressed all concerns in the detailed responses to each reviewer. We summarize the main changes here.

(1) We have provided additional experimental results in the Appendix of our manuscript (Section A.8) on diverse types of (such as inted, coated glass, frosted) glass surfaces, shown in Fig. 24, to demonstrate the generalization ability of our model.

(2) We clarify the dataset scope and bias in the manuscript. We further provide a discussion with experimental results of our model on curved glass in the Appendix (Section A.9, Fig. 25).

(3) We have investigated the robustness of our model on the dynamic scenes. The results are provided in the Appendix (Section A.10).

(4)We have investigated the impact of misalignment on the network performance in the Appendix (Section A.6) and provide additional qualitative results in Fig. 23.

(5) We have evaluated the Retinex decomposition through diverse results shown in Fig.~17 of the Appendix.

(6) We have improved the readability by moving the training strategy (Subsection 4.4 "Training Strategy and Loss Functions") and discussion on failure cases (Subsection 5.4 "Limitations") from the Appendix to the main paper.

Assigned Action Editor: ~Yuheng_Jia1

Submission Number: 7580