LLM-guided Hierarchical Search for End-to-end Reasoning Intensive Retrieval

TMLR Paper9193 Authors

Abstract: Search systems are increasingly used for *reasoning-intensive* queries, where what makes a document relevant requires understanding or reasoning over the query–document relation rather than relying on surface vocabulary or topical similarity. The standard recipe - a cheap embedding-based retriever followed by an LLM verifier - works only when the embedding model places the right documents in its top-*k*, an assumption that recent reasoning-intensive IR benchmarks show often fails to hold even for SOTA embedding models. Many recent works propose query-side fixes such as query rewriting and agentic loops, which have shown promise in bringing LLM reasoning to bear on the search process but remain brittle to the embedding model's effectiveness and to the LLM's ability to rewrite the query from its parametric knowledge alone. In this paper, we explore a different paradigm - *LLM-guided hierarchical search* - in which an LLM interacts with the corpus directly via a hierarchically navigable search index, with no embedding model in the loop at search time. We propose **LATTICE**, an instantiation of this paradigm with two technical contributions: 1. a top-down construction of the search index using LLM judgements over multi-level document summaries; and 2. a robust LLM-guided hierarchical search algorithm that mitigates noisy, context-dependent LLM scores via cross-branch reference nodes and path-aggregated latent scores. Through extensive experiments on the reasoning-intensive BRIGHT benchmark, base LATTICE with an off-the-shelf LLM achieves 46.7 nDCG@10 (matching the best fine-tuned ensemble baseline overall). A lightweight ensemble LATTICE++ that fuses LATTICE with cheap retrieval reaches **49.1 nDCG@10**. A controlled same-LLM comparison against sliding-window reranking shows that reranking offers a better tradeoff at low LLM token budgets, but after a moderate token budget LATTICE converges to a higher asymptote. We further show that LATTICE works with open-weight LLMs and remains competitive on traditional IR benchmarks (NQ, SciFact, SciDocs).

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Arya_Mazumdar1

Submission Number: 9193