Accelerating Federated Split Learning via Local-Loss-Based Training

Dong-Jun Han, Hasnain Irshad Bhatti, Jungmoon Lee, Jaekyun Moon

Keywords: Federated Learning, Split Learning

Abstract: Federated learning (FL) operates based on model exchanges between the server and the clients, and suffers from significant communication as well as client-side computation burden. Emerging split learning (SL) solutions can reduce the clientside computation burden by splitting the model architecture between the server and the clients. However, SL-based ideas still require significant time delay, since each participating client should wait for the backpropagated gradients from the server in order to update its model. Also, the communication burden can still be substantial, depending on various factors like local dataset size and shape of cut layer activations/gradients. In this paper, we propose a new direction to FL/SL based on updating the client/server-side models in parallel, via local-loss-based training specifically geared to split learning. The parallel training of split models substantially shortens latency while obviating server-to-clients communication. We provide latency analysis that leads to optimal model cut as well as general guidelines for splitting the model. We also provide a theoretical analysis for guaranteeing convergence and understanding interplay among different hyperparameters and system constraints. Extensive experimental results indicate that our scheme has significant communication and latency advantages over existing FL and SL ideas.

One-sentence Summary: We propose a fast and communication-efficient algorithm that provides a new direction to federated/split learning, via local-loss-based training highly tailored to split learning.