A Game Theoretical Framework for Inter-process Adversarial Intervention Detection

Muhammed O. Sayin, Hossein Hosseini, Radha Poovendran, Tamer Basar

Abstract: In this paper, we propose and analyze a two-level game theoretical framework to detect advanced and persistent threats across processes. The two-level framework adapted facilitates abstraction of the complexity of process level interactions between defense mechanisms and adversaries from easier to interpret and more flexible system-level interaction. At the process-level, program anomaly detection algorithms have already been proposed to detect anomalous program behavior by comparing monitored activities with the predetermined expected behavior. This had led to significant detection performance initially until advanced adversaries modified the attacks to remain undetected. Therefore, we propose defense mechanisms that anticipate the reaction of advanced evaders and seek to maximize the complexity of undetectable attacks at the expense of additional false alarm rate. Furthermore, in the system-level, we propose defense mechanisms to detect adversarial intervention across processes through the assessment of all process activities together in a cohesive way so that the advanced adversaries need to craft their attacks further to remain undetected also at the system-level. This further increases the cost of complexity for the attacker, and correspondingly degrades the motivation to attack. We provide a game theoretical incentive analysis for both defenders and adversaries, and characterize pure and mixed strategy equilibria. We also analyze the coupling between the two levels of the game.