Concept drift detection based on equal density estimation

Feng Gu, Guangquan Zhang, Jie Lu, Chin-Teng Lin

Abstract: An important problem that remains in online data mining systems is how to accurately and efficiently detect changes in the underlying distribution of large data streams. The challenge for change detection methods is to maximise the accumulative effect of changing regions with unknown distribution, while at the same time providing sufficient information to describe the nature of the changes. In this paper, we propose a novel change detection method based on the estimation of equal density regions, with the aim of overcoming the issues of instability and inefficiency that underlie methods of predefined space partitioning schemes. Our method is general, nonparametric and requires no prior knowledge of the data distribution. A series of experiments demonstrate that our method effectively detects concept drift in single dimension as well as high dimension data, and is also able to explain the change by locating the data points that contribute most to the change. The detection result is guaranteed by statistical tests.