Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve

Download PDF
Open Webpage

Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, Ramachandran Ramjee

Published: 01 Jan 2024, Last Modified: 23 Mar 2025OSDI 2024EveryoneRevisionsBibTeXCC BY-SA 4.0