Comparative Performance Analysis of Lightweight BERT-derived Models for Cybersecurity Corpus in Cloud Networks

Yue Wu, Lin Ye, Baisong Li, Hongli Zhang, Jing Cao

Abstract: The rapid development of cloud networks has created a complex cybersecurity landscape, with new challenges and threats arising from the shift to cloud-based systems. Efficient on-device processing of network security reports has become essential for handling the distributed and edge-oriented scenario with substantial data volumes. Pre-trained models, like BERT, have been extensively used in network security, but their high computational resource demands and limited input length (512 tokens) can hinder their performance in cloud network environments, particularly when classifying network security reports that often exceed this limit. Consequently, this paper presents a comparative analysis of the performance of a series of lightweight pre-trained models derived from BERT on cybersecurity corpus. In order to solve the limitation of the input length of the BERT-derived model, we leverage the latest research advancements in the Natural Language Processing (NLP) domain, adopting an approach that involves abstract extraction, application of pre-trained models, and fine-tuning, to accomplish the task of classifying network security reports on edge devices. A series of comparative experiments conducted on a real-world dataset revealed that employing the BRIO-TinyBERT-Fine-tuning architecture for network security report classification achieved an accuracy rate of 82%. Remarkably, this model utilized only half the parameters of the standard BERT model.