Missingness-aware prompting for modality-missing RGBT tracking

Guyue Hu, zhanghuan Wang, Chenglong Li, Duzhi Yuan, Bin He, Jin Tang

Abstract: RGBT tracking has drawn great attention recently due to its ability to leverage enhancement and complementary information from the RGB and thermal infrared modalities. Nevertheless, RGBT tracking in real-world scenarios inevitably encounters heavy modality-missing challenges caused by substantial environmental factors (such as device overheating, and frame skipping). Existing methods for RGBT tracking are built upon pre-processed missingness-free datasets and suffer significant performance degradation when applied to noisy datasets with random missing modalities. In this paper, we propose a novel missingness-aware prompting framework (MAP) for modality-missing RGBT tracking. It is a lightweight prompting framework consisting of two-stage prompts focusing on compensating essential information for RGBT tracking stage-bystage. Specifically, prototypical missingness-aware prompts (pMAP) are explored to compensate for modality-specific but instance-agnostic prototypical missing information. Contextual missingness-aware prompts (cMAP) are further designed to compensate for instance-specific detailed missing information. Extensive experiments on three large-scale datasets demonstrate the effectiveness and superiority of the proposed framework for RGBT tracking with random missing modalities.