Honey: Harmonizing Progressive Federated Learning via Elastic Synergy across Different Training Blocks
Keywords: Memory-Heterogeneous Federated Learning, Progressive Training, On-Device Training
Abstract: Memory limitation is becoming the prevailing challenge that hinders the deployment of Federated Learning on mobile/IoT devices in real-world cases. Progressive training offers a promising alternative to surpass memory constraints. Instead of updating the full model in each training round, progressive training divides the model into multiple blocks and iteratively updates each block until the full model is converged. However, existing progressive training approaches suffer from prominent accuracy degradation as training each block in isolation drives it to prioritize features that are only beneficial to its specific needs, neglecting the overall learning objective. To address this issue, we present $\texttt{\textbf{Honey}}$, a synergistic progressive training approach that integrates the holistic view and block-wise feedback to facilitate the training of each block. Specifically, the holistic view broadens the learning scope of each block, ensuring that it operates in harmony with the global objective and benefits the training of the whole model. Simultaneously, block-wise feedback heightens each block's awareness of its role and position within the full model, empowering it to make real-time adjustments based on insights from downstream blocks and facilitating a smooth and consistent information flow. Furthermore, to fully harness the heterogeneous memory resources of participating devices, we develop an elastic resource harmonization protocol.
This protocol authorizes each device to adaptively train specific layers according to their memory capacity, optimizing resource utilization, sparking cross-block communication, and accelerating model convergence. Comprehensive experiments on benchmark datasets and models demonstrate that $\texttt{\textbf{Honey}}$ outperforms state-of-the-art approaches, delivering an exceptional average accuracy improvement of up to 43.9\%. Moreover, $\texttt{\textbf{Honey}}$ achieves comparable performance even with a reduction in peak memory usage of up to 49\%.
Primary Area: other topics in machine learning (i.e., none of the above)
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 1061
Loading