BotCensor: Detecting DGA-Based Botnet Using Two-Stage Anomaly Detection

Biao Qi, Jianguo Jiang, Zhixin Shi, Rui Mao, Qiwen Wang

Abstract: Nowadays, most botnets utilize domain generation algorithms (DGAs) to build resilient and agile command and control (C&C) channels. Specifically, botmasters employ DGAs to dynamically produce a large number of random domains and only register a small subset for their actual C&C servers with the purpose to defend them from takeovers and blacklisting attempts. While many approaches and models have been developed to detect DGA-based botnets, they suffer from several limitations, such as difficulties of DNS traffic collection, low feasibility and scalability, and so forth. In this paper, we present BotCensor, a new system that can determine if a host is infected with certain DGA malware with two-stage anomaly detection. In the first stage, we preliminarily attempt to identify malicious domains using a Markov model, and in the second stage, we re-examine the hosts that requested aforementioned malicious domains using novelty detection algorithms. Our experimental results show that our approach performs very well on identifying previously unknown DGA-generated domains and detects DGA bots with high efficiency and efficacy. Our approach not only can be regarded as security forensics tools, but also can be used to prevent malware infections and spread.