Compact-index: an efficient index algorithm for network traffic

Yue Wang, Guangxing Zhang, Haiyang Jiang, Gaogang Xie

Abstract: In many network security systems, network packets will be archived with no loss for the purpose of forensic, troubleshooting and so on. In order to achieve fast retrieval for these stored packets, index is essential. However, with the rapid increase of network link bandwidth, indexing network traffic traces is facing the challenges of index construction speed, index storage overhead and retrieval efficiency. In this paper, we propose an efficient index scheme for packet index, named Compact-Index, that not only effectively reduces the storage cost of index, but also greatly improves index insertion rate and retrieval efficiency. Experimental results show that our scheme can achieve 7.14Mpps index insertion rate for IPv4 traffic and 6.11Mpps index insertion rate for IPv6 traffic, and supports millisecond scale response to most queries. In addition, the average space cost of index is only about 5% of the raw network traffic. All the experimental results above indicate that Compact-Index significantly outperforms the existing state-of-art index schemes.