Learning-Based Frequency Estimation Algorithms

Chen-Yu Hsu, Piotr Indyk, Dina Katabi, Ali Vakilian

Sep 27, 2018 ICLR 2019 Conference Blind Submission readers: everyone Show Bibtex
  • Abstract: Estimating the frequencies of elements in a data stream is a fundamental task in data analysis and machine learning. The problem is typically addressed using streaming algorithms which can process very large data using limited storage. Today's streaming algorithms, however, cannot exploit patterns in their input to improve performance. We propose a new class of algorithms that automatically learn relevant patterns in the input data and use them to improve its frequency estimates. The proposed algorithms combine the benefits of machine learning with the formal guarantees available through algorithm theory. We prove that our learning-based algorithms have lower estimation errors than their non-learning counterparts. We also evaluate our algorithms on two real-world datasets and demonstrate empirically their performance gains.
  • Keywords: streaming algorithms, heavy-hitters, Count-Min, Count-Sketch
  • TL;DR: Data stream algorithms can be improved using deep learning, while retaining performance guarantees.
0 Replies