Lightweight Model Inference on Resource-Constrained Computing Nodes in Intelligent Surveillance Systems

Zhuohang Wang, Yunfeng Zhao, Yong Wang, Li Yan, Zhicheng Liu, Chao Qiu, Xiaofei Wang, Qinghua Hu

Abstract: Intelligent Surveillance System (ISS) is an important application combining deep learning with IoT technologies. Meanwhile, multiple targets multiple camera tracking (MTMCT) has been widely recognized as a promising solution for ISS. However, current terminal devices have limited memory, power and computing power. Deep learning models deployed on these resource-constrained devices need to maintain a low-level number of parameters while ensuring inference delay and accuracy as much as possible. In this paper, we propose a lightweight model inference approach for resource-constrained edge computing nodes, which leverages the limited computing capability of edge devices for collaborative processing of model inference tasks. We consider a system model that includes a combination of both horizontal and vertical methods to dynamically partition deep neural networks. Further, we propose an adaptive strategy that dynamically controls the execution part in the terminal devices. In addition, a learning-based algorithm is used to obtain the near-optimal solutions for dynamic resource allocation decisions. Finally, we evaluate the system in terms of two metrics, i.e., computational latency and network throughput, based on a real-world dataset of campus surveillance. The experimental results reveal the superior effectiveness of the proposed scheme.