OSMO: Enhanced Offloading for Data Stream Perception with Smoothness and Orderliness

Chaokun Zhang, Quan Fan, Jinlong E, Shen He

Abstract: On-device AI is taking over our daily lives by moving closer to mobile devices as perception applications. A data stream perception application generally has three essential requirements: timeliness, smoothness, and orderliness. Most researchers’ efforts to date have proposed various offloading approaches to accelerate compute-intensive AI algorithms in perception applications, thereby fulfilling the requirement of timeliness. However, the lack of concern about the smoothness and orderliness of the data stream will result in fluctuation and commotion anomalies that greatly impair the user experience. In this paper, we propose an enhanced Offloading System with sMoothness and Orderliness (OSMO) to guarantee perception applications’ smooth refresh rates while processing data streams in proper orders with low overhead. OSMO takes advantage of heterogeneous computing devices and data-level parallelism in the offloading process. A scheduling strategy is further devised that dynamically tunes a set of parameters to achieve the best trade-offs among the three requirements of perception applications. We implement a prototype system based on TensorFlow and its typical Android demos. Real-world evaluations demonstrate that our solution can effectively address the fluctuation and commotion issues while providing a high data processing rate with multi-device collaboration.