Evaluating Cyber Threat Intelligence: Accuracy, Completeness, Relevance, and Freshness

Asad Ali, Ren-Hung Hwang, Ying-Dar Lin

Abstract: In the realm of cybersecurity, the extraction of Cyber Threat Intelligence (CTI) is vital for acquiring accurate Tactics, Techniques, and Procedures (TTPs) and Indicators of Compromise (IoC) from reputable sources. The extracted CTI must be evaluated to ensure high quality, and different CTI platforms, such as VirusTotal, AlienVault, and MetaDefender, often yield varying evaluations. A robust method is required that automatically extracts CTI, evaluates it, and provides a verdict on IoCs, determining whether they are malicious, while also evaluating IoC information across multiple platforms. In this work, we propose an automated mechanism that first extracts TTPs and IoCs from reputable threat reports, submits the extracted CTI to multiple platforms, evaluates the platform responses using four key metrics —accuracy, freshness, completeness, and relevance, and provides weighted verdicts for IoCs. We tested 600 IoCs in February 2025, and the weighted verdict matched those of VirusTotal for 79.4%, AlienVault for 87.4%, and MetaDefender for 39.6% of the IoCs. The results also show that VirusTotal provides a consistent evaluation of various types of IoCs in terms of freshness, completeness, and relevance of information, whereas AlienVault shows inconsistencies across all IoC types, and MetaDefender shows inconsistency for some. VirusTotal also outperforms the other two when it comes to providing fresher and more complete intelligence, while AlienVault provides the most relevant information in terms of Structured Threat Information eXpression (STIX) 2.1 objects.

External IDs:dblp:journals/access/AliHL25