Lean Clients, Full Accuracy: Hybrid Zeroth- and First-Order Split Federated Learning

ICLR 2026 Conference Submission21524 Authors

Keywords: federated split learning, resource-efficient split learning, zeroth-order optimization, low rank, fine-tuning

Abstract: Split Federated Learning (SFL) enables collaborative training between resource-constrained edge devices and a compute-rich server by partitioning deep neural networks. Communication overhead is a central issue in SFL and is well mitigated with auxiliary networks; yet the core client-side computation challenge remains, as back-propagation requires substantial memory and computation costs, severely limiting the scale of models that edge devices can support. To make the client side more resource-efficient, we propose HERON-SFL, a novel hybrid optimization framework that integrates zeroth-order (ZO) optimization for local client training while retaining first-order (FO) optimization on the server. With the assistance of auxiliary networks, ZO updates enable clients to approximate local gradients using perturbed forward-only evaluations per step, eliminating memory-intensive activation caching and avoiding explicit gradient computation in the traditional training process. Leveraging the low effective rank assumption, we theoretically prove that HERON-SFL's convergence rate is independent of model dimensionality, addressing a key scalability concern common to ZO algorithms. Empirically, on ResNet training and large language model (LLM) fine-tuning tasks, HERON-SFL matches benchmark accuracy while reducing client peak memory by up to 64\% and client-side compute cost by up to 65\% per step, substantially expanding the range of models that can be trained or adapted on resource-limited devices.

Primary Area: optimization

Submission Number: 21524