Fully Asynchronous Federated Learning with Faster Convergence for LLM Reasoning

Jingyuan Zheng, Siyu Li

Track: long paper (up to 10 pages)

Keywords: Federated Learning, Large Language Model, fine-tuning

Abstract:

Federated Learning (FL) has emerged as a transformative paradigm for distributed machine learning, enabling collaborative model training across decentralized devices while preserving data privacy. Concurrently, the advent of Large Language Models (LLMs)—such as GPT, Claude, and Qwen—has redefined natural language understanding and generation. Despite their potential, integrating LLMs into FL frameworks remains challenging; conventional synchronous FL mechanisms frequently suffer from significant communication overhead and idle "straggler" delays, exacerbated by the vast parameter space of LLMs and the inherent hardware heterogeneity of edge devices.
To mitigate these inefficiencies, we propose a novel fully asynchronous FL framework specifically optimized for LLM fine-tuning. Our core contribution is a systematic exploration of matrix decomposition and approximation techniques to identify the most effective linear algebraic methods for distributed optimization in asynchronous settings. We evaluate three distinct approaches—Principal Component Analysis (PCA), QR Decomposition with Column Pivoting (QRCP), and CUR Decomposition—through extensive experiments on GPT-2 fine-tuning using the WikiText dataset. Empirical results demonstrate that PCA-based approximation achieves the fastest convergence and competitive accuracy, significantly reducing wall-clock training time while maintaining a performance profile comparable to synchronous baselines.

Presenter: ~Jingyuan_Zheng3

Format: Yes, the presenting author will attend in person if this work is accepted to the workshop.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Funding: Yes, the presenting author of this submission falls under ICLR’s funding aims, and funding would significantly impact their ability to attend the workshop in person.

Submission Number: 156