SoK: Evaluations in Industrial Intrusion Detection Research

JSYS 2023 Aug Papers Submission2 Authors

21 Jul 2023 (modified: 18 Oct 2023)JSYS 2023 Aug Papers SubmissionEveryoneRevisions
Keywords: Industrial Intrusion Detection, IDS, Evaluation, Metrics
TL;DR: We surveyed 609 publications proposing Industrial Intrusion Detection Systems and analyzed how evaluations are performed in that research domain, unveiling several shortcomings.
Abstract: Industrial systems are increasingly threatened by cyberattacks with potentially disastrous consequences. To counter such attacks, industrial intrusion detection systems strive to timely uncover even the most sophisticated breaches. Due to its criticality for society, this fast-growing field attracts researchers from diverse backgrounds, resulting in 130 new detection approaches in 2021 alone. This huge momentum facilitates the exploration of diverse promising paths but likewise risks fragmenting the research landscape and burying promising progress. Consequently, it needs sound and comprehensible evaluations to mitigate this risk and catalyze efforts into sustainable scientific progress with real-world applicability. In this paper, we therefore systematically analyze the evaluation methodologies of this field to understand the current state of industrial intrusion detection research. Our analysis of 609 publications shows that the rapid growth of this research field has positive and negative consequences. While we observe an increased use of public datasets, publications still only evaluate 1.3 datasets on average, and frequently used benchmarking metrics are ambiguous. At the same time, the adoption of newly developed benchmarking metrics sees little advancement. Finally, our systematic analysis enables us to provide actionable recommendations for all actors involved and thus bring the entire research field forward.
Area: Real-time and Cyber-physical Systems
Type: Systemization of Knowledge (SoK)
Revision: Yes
Previous Version:
Submission Number: 2