Modality adversarial neural network for visible-thermal person re-identification

Yi Hao, Jie Li, Nannan Wang, Xinbo Gao

Abstract: Highlights • We propose a modality adversarial learning strategy, making our model aligns the feature representations across different modalities effectively. The adversarial learning strategy can competently eliminate modality discrepancy, which enhances the feature representations for cross-modality identification problem. • Our proposed dual-constrained triplet loss can address the intra- and cross- modality variations effectually. We integrate the dual-constrained triplet loss and the identity loss to enhance the discriminative property of feature representations and to generate the modality-invariant features with adversarial training strategy. • We design a one-stream framework for visible-thermal person re-identification to utilize our adversarial learning strategy and dual-constrained triplet loss. Compared with two-stream networks, the one-stream framework is more stable for adversarial learning. Our extension experiments and analysis abundantly prove that the proposed method outperforms most of the state-of-the-art algorithms for visible-thermal person re-identification. Abstract Existing Visible-Thermal Person Re-identification (VT-REID) methods usually adopt two-stream networks for cross-modality images. The two streams are trained to extract features from different modality images respectively. In contrast, we design a Modality Adversarial Neural Network (MANN) to solve VT-REID problem. Our proposed MANN includes a one-stream feature extractor and a modality discriminator. The heterogeneous images are processed by the feature extractor to generate modality-invariant features. And the designed modality discriminator aims to distinguish whether the extracted features are from visible or thermal modality. Moreover, our advanced dual-constrained triplet loss is introduced for better cross-modality matching performance. The experiments on two cross-modality person re-identification datasets show that MANN can effectively learn modality-invariant features and outperform state-of-the-art methods by a large margin.