TaiChi: Efficient Execution for Multi-DNNs Using Graph-Based Scheduling

Xilang Zhou, Haodong Lu, Tianchen Wang, Zhuoheng Wan, Jianli Chen, Jun Yu, Kun Wang

Abstract: Applications constructed with multiple Deep Neural Networks (multi-DNNs) are growing rapidly in edge and data center. However, executing multi-DNNs efficiently remains chal-lenging because multi-DNNs are inherently heterogeneous. The diverse operators, dependencies and performance requirements of multi-DNNs lead to high costs of encoding and generalization. We introduce Taichi, a graph-based framework for efficiently scheduling multi-DNNs on multi-core accelerators. Specifically, Taichi consists of two phases: (1) a graph neural network (GNN) is utilized to automatically capture the features from the graph structure of multi-DNNs and (2) reinforcement learning (RL) is employed to find an optimal online scheduling strategy. Evaluation results show that TaiChi reduces latency by 1.1-2.4 x and 1.1-1.6x compared to SJF and MAGMA, and improves throughput by 26.4-63.7% and 18.6-33.7%, respectively. Moreover, TaiChi achieves an average speedup of 779 x in scheduling runtime compared to MAGMA.