Multimodal Pseudo-Labeling Under Various Shooting Conditions: Case Study on RGB and IR Images

Hiroki Kojima, Naoshi Kaneko, Seiya Ito, Kazuhiko Sumi

Abstract: In recent years, large-scale datasets with accurate labels have been an extremely important factor in the progress of computer vision. One typical example is object detection in outdoor scenes, where data captured under various conditions such as lighting, weather, and temperature are essential to increase the robustness of object detectors. However, such is time-consuming. In addition, under conditions such as extremely low luminance, it is difficult to assign accurate labels, even manually. When even a small amount of labeled data is available, pseudo-labeling can be used to effectively assign labels to unlabeled data, but if the labeled data is captured under only a single condition (e.g., daytime), it is difficult to perform pseudo-labeling to images under different conditions (e.g., nighttime). In this paper, we propose a pseudo-labeling method under various conditions by using multimodal images. If one of which has a small change in texture depending on the conditions and the other has a large change, we can perform pseudo-labeling and self-training by projecting the outputs mutually. Using videos taken by RGB and IR cameras on a road as a case study, we show the effectiveness of the proposed method in object detection.