A Multi-Level Feature Distribution Learning Method for Automatic Modulation Open-Set Recognition

zhenxi zhang

Abstract: The study of open-set recognition for modulation types in communication signals is of high research significance, as it addresses critical challenges in wireless communication systems, such as spectrum monitoring, interference identification, and secure transmission. Traditional closed-set recognition methods are limited to pre-defined modulation types, which restrict their adaptability in real-world environments where new or unknown modulation types may emerge. In contrast, Automatic Modulation Open-Set Recognition (AMOSR) not only classifies known modulation types but also explicitly recognizes unknown ones, thereby addressing the inherent limitations of closed-set approaches. However, existing AMOSR methods face challenges in achieving a balanced optimization between the empirical risk for known modulation types and the open space risk for unknown modulation types. This issue results in two adverse effects: (1) low rejection accuracy for unknown modulation signals, and (2) degraded classification performance on known types due to overfitting to the closed-set training data. To address these issues, we propose a multi-level feature distribution learning (MLFDL) method for AMOSR, which integrates a pseudo modulation placeholder method with feature distribution constraints. To simulate unknown modulation types during training, we introduce a pseudo modulation placeholder via manifold mixup, which constructs synthetic samples in the feature space to approximate the behavior of unseen modulations. We design a multi-level feature distribution constraint method, combining sample-centroid contrastive learning with a max-min feature constraint, ensuring adequate feature space for unknown modulation signals and enhancing the separation between known and unknown signals. Comprehensive experiments across different datasets show that MLFDL achieves state-of-the-art performance on both AUROC and OSCR metrics, with minimum sample-to-centroid distance serving as a robust decision criterion for unknown signal recognition.