MultiTASC: A Multi-Tenancy-Aware Scheduler for Cascaded DNN Inference at the Consumer Edge

Sokratis Nikolaidis, Stylianos Venieris

Keywords: Vision and Learning, Other, Artificial Intelligence, Deep Learning, Edge Computing, Distributed Systems, Mobile Computing

TL;DR: We propose a novel scheduling scheme that enables a single edge server to efficiently support multiple heterogeneous edge devices in a distributed cascade architecture.

Abstract:

Cascade systems comprise a two-model sequence, a lightweight model processing all samples and a heavier model conditionally refining harder samples to improve accuracy. By placing the light model on the device side and the heavy model on a server, model cascades constitute a widely used distributed inference approach. With the rapid expansion of intelligent indoor environments, the new setting of Multi-Device Cascade is emerging where multiple and diverse devices are simultaneously using a shared heavy model on the same server, typically located close to the consumer environment. This work presents MultiTASC, a multi-tenancy-aware scheduler that adaptively controls the forwarding decision functions of the devices in order to maximize the system throughput, while sustaining high accuracy and low latency. By explicitly considering device heterogeneity, our scheduler improves the latency service-level objective (SLO) satisfaction rate over state-of-the-art cascade methods in highly heterogeneous setups, while serving over 40 devices, showcasing its scalability.

Keywords: Vision and Learning, Other

Published in proceedings of 2023 IEEE Symposium on Computers and Communications (ISCC) and received the Best Student Full Paper Award, link: https://ieeexplore.ieee.org/document/10217872

Reference: S. Nikolaidis, S. I. Venieris and I. S. Venieris, "MultiTASC: A Multi-Tenancy-Aware Scheduler for Cascaded DNN Inference at the Consumer Edge," 2023 IEEE Symposium on Computers and Communications (ISCC), Gammarth, Tunisia, 2023, pp. 411-416, doi: 10.1109/ISCC58397.2023.10217872.

Submission Number: 102