PSscheduler: A parameter synchronization scheduling algorithm for distributed machine learning in reconfigurable optical networks

Ling Liu, Xiaoqiong Xu, Pan Zhou, Xi Chen, Daji Ergu, Hongfang Yu, Gang Sun, Mohsen Guizani

Abstract: With the increasing size of training datasets and models, parameter synchronization stage puts a heavy burden on the network, and communication has become one of the main performance bottlenecks of distributed machine learning (DML). Concurrently, optical circuit switch (OCS) with high bandwidth and reconfigurable features has increasingly introduced into the construction of network topology, obtaining the reconfigurable optical networks. Actually, OCS is conducive to accelerating the parameter synchronization stage, and thus improves training performance. However, unreasonable circuit scheduling algorithm has a great impact on parameter synchronization time because of non-negligible OCS switching delay. Besides, most of the existing circuit scheduling algorithms do not effectively use the training characteristics of DML, and the performance gains are limited. Therefore, in this paper, we study the parameter synchronization scheduling algorithm in reconfigurable optical networks, and propose PSscheduler by jointly optimizing the circuit scheduling and deployment of parameter servers in parameter server (PS) architecture. Specifically, a mathematical optimization model is established first, which takes into account the deployment of parameter servers, the allocation of parameter blocks and circuit scheduling. Subsequently, the mathematical model is solved by relaxed variables and deterministic rounding approach. The results of simulation based on real DML workloads demonstrate that compared to Sunflow and HLF , PSscheduler is more stable and can reduce parameter synchronization time (PST) by up to 46.61% and 25%, respectively.