Adaptive Denoising for Network Traffic Measurement

He Huang, Chen Lou, Yu-e Sun, Yang Du, Shigang Chen, Guoju Gao, Hongli Xu

Abstract: Traffic measurement in high-speed networks is crucial for applications like traffic engineering, network management, and surveillance. Restricted by the limitations of on-chip memory resources and the speed of packet processing, most existing solutions use compact data structures, namely sketches, to facilitate line-speed measurement. Nevertheless, these sketches, due to their shared record units (bits/counters) among flows, inevitably introduce noise into the measurement result of each flow. While conventional average denoising strategies can mitigate noise from raw estimates, they fall short of providing sufficient accuracy for medium-sized flows, primarily due to the uneven distribution of noise. To complement prior work, we propose two algorithms, ADN and mADN, which can perform denoising by considering the sizes of shared flows. ADN employs an optimization algorithm to model interconnections among flows, thereby reconstructing noise propagation and accurately restoring their sizes. Meanwhile, mADN retains the benefits of ADN yet excels in being more memory-efficient and precise. We apply our estimators to five essential tasks: per-flow size estimation, heavy hitter detection, heavy change detection, distribution estimation, and entropy estimation. Experimental results based on real Internet traffic traces show that our measurement solutions surpass existing state-of-the-art approaches, reducing the mean absolute error by approximately an order of magnitude under the same on-chip memory constraints.