When foresight pruning meets zeroth-order optimization: Efficient federated learning for low-memory devices

Pengyu Zhang, Yingjie Liu, Yingbo Zhou, Xian Wei, Mingsong Chen

Abstract: To facilitate Federated Learning (FL) on low-memory embedded devices, various federated pruning methods aim to reduce memory usage during inference but have a limited impact on training memory burdens. Alternatively, zeroth-order or backpropagation-free (BP-Free) methods can partially alleviate memory consumption but still face computational overhead as the number of model parameters increases. To address these issues, we propose a memory-efficient federated foresight pruning method based on the Neural Tangent Kernel (NTK), which seamlessly integrates with federated BP-Free training frameworks. We approximate federated NTK using local NTK matrices and demonstrate that the data-free property of our method significantly reduces approximation error in highly heterogeneous data scenarios. Our method improves the vanilla BP-Free method with fewer floating point operations (FLOPs) and alleviates memory pressure during pruning and training, making FL more feasible for low-memory devices. Experimental results on simulation- and real test-bed-based platforms show that our method improves the accuracy by up to 6.35% and reduces the FLOPs by up to 57% against the vanilla BP-Free method while maintaining the same 9×<math><mrow is="true"><mn is="true">9</mn><mo is="true">×</mo></mrow></math> memory usage saving.

External IDs:dblp:journals/jsa/ZhangLZWC25