SPD: Sync-Point Drop for efficient tensor parallelism of Large Language Models
Keywords: sync point drop, tensor parallelism, distributed inference, efficient ml
Abstract: With the rapid expansion in the scale of large language models (LLMs), enabling efficient distributed inference across multiple computing units has become increasingly critical. However, communication overheads from frequent synchronization during distributed inference pose a significant challenge to achieve scalability and low latency. Therefore, we introduce a novel optimization technique, Sync-Point Drop (SPD) to reduce communication overheads in tensor parallelism by dropping synchronization on attention outputs. In detail, we first propose a block design that allows execution to proceed without communication through SPD. Second, we identify regions of communication redundancy, where dropping synchronization results in no loss of model performance. In addition, to extend SPD across all compute blocks, we employ a low-cost distillation, specifically targeting blocks giving quality degradation, to maximize accuracy recovery. For extreme blocks where performance degradation is severe, we introduce a new head grouping enhancements to amplify the distillation’s recovery effect. The proposed methods effectively alleviate communication bottlenecks while minimizing accuracy degradation during LLM inference, offering a scalable solution for distributed environments.
Primary Area: optimization
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 12793
Loading