An Efficient IOC-Driven BigData Tracing and Backtracking Model for Emergency Response

Haiping Wang, Jianqiang Li, Binbin Li, Tianning Zang, Zisen Qi, Siyu Jia, Yu Ding, Yifei Yang

Abstract: In the realm of intelligence-driven emergency response, utilizing Indicators of Compromise (IOCs) like IP addresses and domain names is crucial for the swift analysis of and response to cyber-attacks within vast network security logs. However, with daily log increases of hundreds of TBs and total volumes reaching into the hundreds of PBs, the tasks of tracking and backtracking critical data face significant challenges, including substantial computational resource consumption and extended processing times. In this paper, we introduce a data tracking and backtracking model based on the principles of data tiered organization and pre-tracking backtracking to enhance emergency response efficiency and reduce resource consumption. The model initially scores IOC clues based on dimensions such as activity and confidence levels, periodically filtering and caching those with higher scores. Based on these clues, it pre-tracks and backtracks associated data, thereby avoiding the computational overhead caused by multiple repetitive offline filtering operations. Furthermore, to improve the retrieval efficiency of threat intelligence clues in scenarios involving massive data volumes, we have designed a three-tier data pre-tracking and backtracking state index to manage the data involved in pre-tracking and backtracking. We also propose a data query task decomposition algorithm to optimize the data retrieval process and enhance emergency response efficiency. Experimental results demonstrate that our method reduces processing time by more than 85.6% compared to traditional direct raw data access, highlighting our model’s real-world efficiency and practicality.