A Distributed Algorithm for Finding Global Icebergs with Linked Counting Bloom Filters

Kui Wu, Yang Xiao, Jie Li, Bo Sun

Abstract: Icebergs denote data items whose total frequency of occurrence is greater than a given threshold. When data items are scattered across a large number of network nodes, searching for global icebergs becomes a challenging task especially in bandwidth limited wireless networks. Existing solutions require a central server for ease of algorithm design and/or use random sampling to reduce bandwidth cost. In this paper, we present a new distributed algorithm to search for global icebergs without any centralized control or random sampling. A new type of Bloom filter, called linked counting Bloom filter, is designed to check the membership of a set and to store the accumulative frequency of data items. We evaluate the performance of our distributed algorithm with real data sets.