Scenario-Guided Transformer-Enabled Multi-Modal Unknown Event Classification for Air Transport

Yang Yang, Yishan Zhang, Shengsheng Qian, Kaiquan Cai

Abstract: With the rapid rise of massive multi-modal Internet data, air crisis event recognition plays a significant role in accident information management. A key feature of this work is the open-world setting, where the new events can be detected via the model trained from the known events. A scenario-guided Transformer-enabled multi-modal unknown air crisis event classification is proposed. Firstly, we introduce a memory-augmented feature representation module to improve the cross-modal feature fusion in the basic Transformer network, where the textual and image features of the air crisis events are extracted through the pre-trained models. Then, a scenario-guided mechanism is proposed to pivot the pseudo event simulation to approximate the distribution of unknown events effectively. Specifically, an end-to-end scenario attention with a cross-event mask module is presented to select the valid samples of pseudo events and filter out the invalid ones. Finally, the unknown event classifier is designed to classify the known classes and recognize the unknown events simultaneously. Moreover, a specialized multi-modal dataset on air crisis events is proposed as a benchmark, named AirCrisisMMD. The extensive experiments are performed on the AirCrisisMMD and CrisisMMD datasets, where the latter is the publicly available multi-modal crisis event dataset. The results verify the superior performance boost of the proposed method compared with the highly related state-of-the-art baselines.

External IDs:dblp:journals/tits/YangZQC24