Decentralized Training of Transformer Models in Heterogeneous Network
Keywords: Distributed Learning, LLM
Abstract: Training large transformer-based models like GPT-4 and Llama3 is prohibitively expensive, often requiring vast resources, such as tens of thousands of GPUs running simultaneously for months. Traditionally, these models are trained in specialized clusters with high-speed, uniform interconnections and computational capabilities, enabling efficient data and pipeline parallelism. However, these clusters are costly, while more affordable GPUs are widely distributed across the globe. Existing approaches, such as Swarm and Dapple, primarily focus on distributed learning across data centers. In this paper, we introduce a novel framework designed to handle heterogeneous devices and unstable communication environments. Our framework employs a hybrid approach, combining parameter server architectures, pipeline parallelism, and task pool strategies to effectively manage device disconnections. Through comprehensive time-cost analysis and graph clustering techniques, we derive a near-optimal resource allocation scheme. We compare our method with existing large-scale training approaches and demonstrate its effectiveness by training a large language model using gaming GPUs in real-world internet conditions.
Primary Area: infrastructure, software libraries, hardware, systems, etc.
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 9467
Loading