Transformer Based Remote Sensing Object Detection With Enhanced Multispectral Feature Extraction

Jiahe Zhu, Xu Chen, Huan Zhang, Zelong Tan, Shengjin Wang, Hongbing Ma

Abstract: As a convention, satellites and drones are equipped with sensors of both the visible light spectrum and the infrared (IR) spectrum. However, existing remote sensing object detection methods mostly use RGB images captured by the visible light camera while ignoring IR images. Even for algorithms that take RGB–IR image pairs as input, they may fail to extract all potential features in both spectrums. This letter proposes Multispectral DETR, a remote-sensing object detector based on the deformable attention mechanism. To enhance multispectral feature extraction and attention, DropSpectrum and SwitchSpectrum methods are further proposed. DropSpectrum facilitates the extraction of multispectral features by requiring the model to detect some of the targets with only one spectrum. SwitchSpectrum eliminates the level bias caused by the fixed order of RGB–IR feature maps and enhances attention to multispectral features. Experiments on the vehicle detection in aerial imagery (VEDAI) dataset show the state-of-the-art performance of Multispectral DETR and the effectiveness of both DropSpectrum and SwitchSpectrum.