An Efficient Transformer Encoder-Based Classification of Malware Using API Calls

Chen Li, Zheng Chen, Junjun Zheng

Abstract: Malware is a major security threat to computer systems and significantly impacts system reliability. Recurrent neural network (RNN)-based methods have attracted much attention in API call-based malware detection in recent decades. However, traditional RNNs have a gradient vanishing problem when processing long API call sequences. This paper proposes a transformer encoder-based model, called MalTransEn, to solve the limitations of RNN. In particular, a novel transformer encoder-based classifier is proposed to classify malware by learning interaction features in sequences of API calls. Experimental results showed that the proposed architecture performed well and outperformed other deep learning-based baselines.