On the Role of Temperature Sampling in Test-Time Scaling

Yuheng Wu, Azalia Mirhoseini, Thierry Tambe

Keywords: Test-Time Scaling, Inference-Time Compute, Scaling Law, Sampling Methods, Evaluation

Abstract: Large language models (LLMs) can improve reasoning at inference time through test-time scaling (TTS), where multiple reasoning traces are generated and the best one is selected. Prior work shows that increasing the number of samples $K$ steadily improves accuracy. In this paper, we demonstrate that this trend does not hold indefinitely: at large $K$, further scaling yields no gains, and certain hard questions remain unsolved regardless of the number of traces. Interestingly, we find that different sampling temperatures solve different subsets of problems, meaning single-temperature scaling explores only part of a model’s potential. We therefore propose scaling along the temperature dimension, which enlarges the reasoning boundary of LLMs. Temperature scaling enables base models to reach performance comparable to reinforcement learning (RL)-trained counterparts, without additional post-training. We further provide a comprehensive analysis of this phenomenon and design a multi-temperature voting method that reduces the overhead of temperature scaling. Overall, our findings suggest that TTS is more powerful than previously thought, and that temperature scaling offers a simple and effective way to unlock the latent potential of base models.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 3834