A Survey on LLM Test-Time Compute via Search: Tasks, LLM Profiling, Search Algorithms, and Relevant Frameworks

TMLR Paper4001 Authors

Abstract: LLM test-time compute (or LLM inference) via search has emerged as a promising research area with rapid developments. However, current frameworks often adopt distinct perspectives on three key aspects—task definition, LLM profiling, and search procedures—making direct comparisons challenging. Moreover, the search algorithms employed often diverge from standard implementations, and their specific characteristics are not thoroughly specified. This survey aims to provide a comprehensive but integrated technical review on existing LIS frameworks. Specifically, we unify task definitions under Markov Decision Process (MDP) and provides modular definitions of LLM profiling and search procedures. The definitions enable precise comparisons of various LLM inference frameworks while highlighting their departures from conventional search algorithms. We also discuss the applicability, performance, and efficiency of these methods.

Submission Length: Long submission (more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=We0DVG0WUl&noteId=We0DVG0WUl

Changes Since Last Submission:

* (Modify Figure 1) Remove references to Sections 2, 3, and 4 to avoid duplication.
* (Add in Section 1) a more detailed background on the capabilities and limitations of LLMs, clarifying why search techniques are essential for effective test-time computation (suggested by Reviewer G9WE)
* (Add in Section 2) Insert new paragraph "Conceptual Foundations of MDPs" and add discussion on embodied tasks and combinatorial tasks.
* (Add in Table 2) Include common benchmarks(, aligning with the suggestion of Reviewer F53r).
* (Add in Section 3) Introduce a stochastic policy and several new evaluators (aligning with the suggestion of Reviewer F53r).
* (Add in Section 5) 7 more highly relevant frameworks in Section 5 (aligning with the suggestion of Reviewer F53r)
* (Add Section 6) for a broader scope of LLM inference + search (aligning with the suggestion of Reviewer F53r)
* (Modify Sections) Reorder the "Related Work" section (currently Section 7) to appear before the "Discussion" section (currently Section 8), so that related work is introduced earlier to be mentioned in the "Discussion".
* (Add in Sections 5, 6, 7) Incorporate 11 new ICLR2025 papers (as suggested by Reviewer F53r).
* (Add in Section 7) Specify fine-tuning frameworks for test-time computation (e.g., ReST).
* (Add in Section 8) Include extra perspectives and future work.
* (Remove in Section 8) Delete the duplicate "Beyond Finite, Fixed Search Space" paragraph.

Assigned Action Editor: ~Yutian_Chen1

Submission Number: 4001