BATON: Enhancing Batch-wise Inference Efficiency for Large Language Models via Dynamic Re-batching
Track: Systems and infrastructure for Web, mobile, and WoT
Keywords: LLM, inference serving, query scheduling
TL;DR: In this paper, we design a tensor shaping and aligning strategy, which can enable seamless batch-wise LLM inference at query level.
Abstract: The advanced capabilities of Large Language Models (LLMs) have inspired the development of various interactive web services or applications, such as ChatGPT, which offer query inference services for users. Unlike traditional DNN model, the inference of LLM entails different iterations of forward computation for different queries, which result in efficiency challenges for existing run-to-completion batch-wise inference. Hence, some methods refine batch-wise inference to iteration-level by duplicating all nonlinear layers of LLM. However, this approach not only increases resource usage but also introduces idle computations to the batch due to the prefilling of newly added queries.
Therefore, we propose BATON, an efficient batch-wise LLM inference scheme by dynamically adjusting processing batch, which can achieve near-zero idle computations without incurring additional resource consumption. To do so, BATON 1) shapes the vectors involved in the inference of the newly inserted query and processing batch to align dimensions and generates a new attention mask based on vector shaping to ensure inference correctness, which enables query inserting without consuming additional resource; 2) embeds prefilled \textit{Keys} and \textit{Values} of the new query into the \textit{KV\_Cache} of the processing batch by leveraging the prefilling and decoding separation mechanism, eliminating idle computations to the batch introduced by the prefilling process of the new query. Experimental results show that compared to the state-of-the-art solution Orca, \name outperforms improves query processing by up to 1.75$\times$.
Submission Number: 967
Loading