LLMShare: Optimizing LLM Inference Serving with Hardware Architecture Exploration

Hongduo Liu, Chen Bai, Peng Xu, Lihao Yin, Xianzhi Yu, Hui-Ling Zhen, Mingxuan Yuan, Tsung-Yi Ho, Bei Yu

Abstract: Large Language Models (LLMs) have revolutionized language tasks but pose significant deployment challenges due to their substantial computational demands during inference. The hardware configurations of existing LLM serving systems do not optimize for the different computational and bandwidth needs of the prefill and decoding phases in LLM inference, leading to inefficient resource use and increased costs. In this paper, we systematically investigate promising hardware configurations for LLM inference serving. We develop a simulator that models the performance and cost across different hardware solutions and introduce a customized design space exploration framework to identify optimal setups efficiently. By aligning hardware capabilities with the specific demands of the prefill and decoding phases, we achieve $13 \%$ cost savings and over $4 \times$ throughput improvements compared to conventional serving system setups.

External IDs:dblp:conf/dac/LiuBXYYZYHY25