Intra-Request Branch Orchestration for Efficient LLM Reasoning

Weifan Jiang, Rana Shahout, Yilun Du, Michael Mitzenmacher, Minlan Yu

Keywords: Efficient LLM Reasoning, Systems for LLM

TL;DR: We reduce the computational cost and latency of reasoning through intra-request orchestration of reasoning branches.

Abstract:

LLMs increasingly rely on inference-time reasoning algorithms such as chain-of-thought and multi-branch reasoning to improve accuracy on complex tasks. These methods, however, significantly increase token usage (cost) and per-request latency.
Prior work has primarily focused on reducing token usage, often at the expense of accuracy, while overlooking other latency factors.

We present DUCHESS, an LLM serving system that reduces computational cost and latency without sacrificing accuracy through intra-request branch orchestration guided by predictions.
Within each request, DUCHESS predicts branch correctness with
a lightweight linear probing model over LLM layer activations. The orchestration policy uses these predictions to decide whether to terminate a branch early, duplicate an existing branch, or continue exploring a branch.
When handling multiple requests, DUCHESS can further reduce latency by prioritizing easier reasoning tasks, when request complexity can be estimated from the prompt.

Experiments on three reasoning benchmarks show that DUCHESS consistently improves the token–accuracy Pareto frontier, reducing token usage by 42–63\% at matched accuracy compared to self-consistency. For request serving with vLLM, DUCHESS reduces mean, median, and tail latencies by 57-81\%, 58-85\%, and 52-84\% with First-Come-First-Served (FCFS) scheduling across three datasets, compared to self-consistency. At higher request rates, scheduling jobs by increasing predicted difficulty reduces latency by 25.1-29.7\% further over FCFS.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 9470