AutoMon: Automatic Distributed Monitoring for Arbitrary Multivariate Functions

Hadar Sivan, Moshe Gabel, Assaf Schuster

Abstract: Approaches for evaluating functions over distributed data streams are increasingly important as data sources become more geographically distributed. However, existing methodologies are limited to small classes of functions, requiring non-trivial effort and substantial mathematical sophistication to tailor them to new functions. In this work we present AutoMon, the first general solution to this problem. AutoMon enables automatic, communication-efficient distributed monitoring of arbitrary functions. Given source code that computes a function from centralized data, the AutoMon algorithm approximates the function over the aggregate of distributed data streams, without centralizing data updates. Our evaluation shows that AutoMon sends the same number or fewer messages as state-of-the-art techniques when monitoring specific functions for which a distributed, hand-crafted solution is known. AutoMon, however, is a lot more powerful. It automatically generates a communication-efficient distributed monitoring solution for arbitrary functions, e.g., monitoring deep neural networks inference tasks for which no non-trivial solution is known.