Day2Dark: Pseudo-Supervised Activity Recognition beyond Silent Daylight

Yunhua Zhang, Hazel Doughty, Cees G. M. Snoek

Abstract: This paper strives to recognize activities in the dark, as well as in the day. As our first contribution, we establish that state-of-the-art activity recognizers are effective during the day, but not trustworthy in the dark. The main causes are the limited availability of labeled dark videos as well as the distribution shift from the lower color contrast. To compensate for the lack of labeled dark videos, our second contribution is to introduce a pseudo-supervised learning scheme, which utilizes unlabeled and task-irrelevant dark videos to improve an activity recognizer in low light. As the lower color contrast results in visual information loss, we propose to incorporate the complementary activity information within audio, which is invariant to illumination. Since the usefulness of audio and visual features differs depending on the amount of illumination, we introduce our `darkness-adaptive' audio-visual recognizer as the third contribution. Experiments on EPIC-Kitchens, Kinetics-Sound, and Charades demonstrate our proposals are superior to image enhancement, domain adaptation and alternative audio-visual fusion methods, and can even improve robustness to occlusions.