Star Attention: Efficient LLM Inference over Long Sequences

Shantanu Acharya, Fei Jia, Boris Ginsburg

TL;DR: Training-free inference technique combining block-sparse and global attention for efficient long-sequence processing, achieving up to 11× speedup while retaining 97–100% accuracy.

Abstract: Inference with Transformer-based Large Language Models (LLMs) on long sequences is both costly and slow due to the quadratic complexity of the self-attention mechanism. We introduce Star Attention, a two-phase block-sparse approximation that improves computational efficiency by sharding attention across multiple hosts while minimizing communication overhead. In the first phase, the context is processed using blockwise-local attention across hosts, in parallel. In the second phase, query and response tokens attend to all prior cached tokens through sequence-global attention. Star Attention integrates seamlessly with most Transformer-based LLMs trained with global attention, reducing memory requirements and inference time by up to 11x while preserving 97-100% of accuracy.

Lay Summary:

Modern AI systems built on large language models (LLMs) are increasingly being used for tasks requiring extremely long inputs, such as analyzing entire codebases or summarizing thousands of pages of documents. But there's a catch: processing such long inputs is slow and expensive because current methods force the model to look at every previous word when generating the next one.

We developed a new method called Star Attention that significantly speeds up this process while maintaining high accuracy. It works by dividing the input into smaller parts and processing them independently across multiple computers. Each part is anchored by a shared section from the beginning, which helps the model stay oriented. Later, when it’s time to generate answers, the model efficiently combines relevant information from all parts without needing to process the full input at once.

This approach makes LLMs up to 11 times faster on long inputs while preserving 97–100% of the original accuracy. It enables more scalable and cost-effective deployment of AI systems for real-world tasks that require understanding and reasoning over very large inputs.

Link To Code: https://github.com/NVIDIA/Star-Attention

Primary Area: Deep Learning->Attention Mechanisms

Keywords: Large Language Models, Transformers, Attention Mechanisms, Sparse Attention, Long Context, Inference Optimization, Distributed Systems

Submission Number: 7740