Data Generation Scheme for Thermal Modality with Edge-Guided Adversarial Conditional Diffusion Model

Guoqing Zhu, Honghu Pan, Qiang Wang, Chao Tian, Chao Yang, Zhenyu He

Abstract:

In challenging low-light and adverse weather conditions, thermal vision algorithms, especially object detection, have exhibited remarkable potential, contrasting with the frequent struggles encountered by visible vision algorithms. Nevertheless, the efficacy of thermal vision algorithms driven by deep learning models remains constrained by the paucity of available training data samples.
To this end, this paper introduces a novel approach termed the edge-guided conditional diffusion model (ECDM). This framework aims to produce meticulously aligned pseudo thermal images at the pixel level, leveraging edge information extracted from visible images.
By utilizing edges as contextual cues from the visible domain, the diffusion model achieves meticulous control over the delineation of objects within the generated images. To alleviate the impacts of those visible-specific edge information that should not appear in the thermal domain, a two-stage modality adversarial training (TMAT) strategy is proposed to filter them out from the generated images by differentiating the visible and thermal modality.
Extensive experiments on LLVIP demonstrate ECDM’s superiority over
existing state-of-the-art approaches in terms of image generation quality.
The pseudo thermal images generated by ECDM also help to boost the performance of various thermal object detectors by up to 7.1 mAP.

Primary Subject Area: [Generation] Generative Multimedia

Secondary Subject Area: [Generation] Generative Multimedia

Relevance To Conference: In challenging low-light and adverse weather conditions, thermal vision algorithms have exhibited remarkable potential, contrasting with the frequent struggles encountered by visible vision algorithms. Nevertheless, the efficacy of thermal vision algorithms driven by deep learning models remains constrained by the paucity of available training data samples. Our manuscript proposed a conditional generative model that generated thermal images from visible images for the problem and proved its effect on thermal object detection. Our work offers valuable insights to the community and provides help to other deep learning-based multispectral tasks, such as RGB-T detection and tracking.

Submission Number: 927