Streaming algorithms for estimating entropy

Nicholas J. A. Harvey, Jelani Nelson, Krzysztof Onak

Abstract: We give a method for estimating the empirical Shannon entropy of a distribution in the streaming model of computation. Our approach reduces this problem to the well-studied problem of estimating frequency moments. The analysis of our approach is based on new results which establish quantitative bounds on the rate of convergence of Renyi entropy towards Shannon entropy.