Abstract: With the advancement of Internet of Things technology, the need for sophisticated signal modulation classification (SMC) has intensified, ensuring seamless communication and bolstering security among interconnected devices. In the contemporary complex channel environment, the difficult lies in dealing with a multitude of modulation schemes that exhibit subtle distinctions. Prior knowledge-guided and deep learning methods have complementary strengths in the current context of SMC. To synthesize the advantages of these two methods, we propose an integrated method of prior knowledge and contrast feature for SMC, called APFS. APFS integrates prior knowledge from the modulation task with feature information acquired through contrastive learning. Feature extraction guided by prior knowledge accurately captures the key patterns in modulated signals. Contrastive learning reveals the inherent distinctions among various modulation modes by comparing different samples. In the joint feature extraction approach for prior knowledge, each form of prior knowledge is first analyzed independently, and then jointed to extract information from its temporal sequence. The contrast features surpass the constraints of labeling and unearth deeper implicit information. In experiments, we systematically compared the performance of our method with various baselines, as well as combinations of prior knowledge and contrast feature. The results demonstrate the superior performance of our method.