TreeHop: Efficient Embedding-Level Query Rewriter

ACL ARR 2025 July Submission989 Authors

Abstract: Retrieval-augmented generation (RAG) systems face significant challenges in multi-hop question answering (MHQA), where complex queries require synthesizing information across multiple document chunks. Existing approaches typically rely on iterative LLM-based query rewriting and routing, resulting in high computational costs due to repeated LLM invocations and multi-stage processes. To address these limitations, we propose TreeHop, an embedding-level framework without the need for LLMs in query refinement. TreeHop dynamically updates query embeddings by fusing semantic information from prior queries and retrieved documents, enabling iterative retrieval through embedding-space operations alone. This method replaces the traditional "Retrieve-Rewrite-Vectorize-Retrieve" cycle with a streamlined "Retrieve-Embed-Retrieve" loop, significantly reducing computational overhead. Moreover, a rule-based stop criterion is introduced to further prune redundant retrievals, balancing efficiency and recall rate. Experimental results show that TreeHop rivals advanced RAG methods across three open-domain MHQA datasets, achieving comparable performance with only 5\%-0.4\% of the model parameter size and reducing the query latency by approximately 99\% compared to concurrent approaches. This makes TreeHop a faster and more cost-effective solution for deployment in a range of knowledge-intensive applications. For reproducibility purposes, codes and data are available anonymously here: https://github.com/Super-Researcher/TreeHop

Paper Type: Long

Research Area: Information Retrieval and Text Mining

Research Area Keywords: Information Retrieval, Retrieval-Augmented Generation, Natural Language Processing, Multi-hop Question Answering, Query Rewrite

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Reproduction study, Approaches to low-resource settings, Approaches low compute settings-efficiency, Publicly available software and/or pre-trained models

Languages Studied: English

Reassignment Request Area Chair: This is not a resubmission

Reassignment Request Reviewers: This is not a resubmission

A1 Limitations Section: This paper has a limitations section.

A2 Potential Risks: Yes

A2 Elaboration: 7

B Use Or Create Scientific Artifacts: Yes

B1 Cite Creators Of Artifacts: Yes

B1 Elaboration: 3.5, 4.1, 4.2

B2 Discuss The License For Artifacts: N/A

B3 Artifact Use Consistent With Intended Use: Yes

B3 Elaboration: 4.1

B4 Data Contains Personally Identifying Info Or Offensive Content: No

B4 Elaboration: The source of adopted datasets are all from Wikipedia.

B5 Documentation Of Artifacts: N/A

B5 Elaboration: Appendix A

B6 Statistics For Data: Yes

B6 Elaboration: 4.1, Appendix B

C Computational Experiments: Yes

C1 Model Size And Budget: Yes

C1 Elaboration: 1, 3.5, 4.2

C2 Experimental Setup And Hyperparameters: Yes

C2 Elaboration: 3.5, 4.2, 4.3

C3 Descriptive Statistics: Yes

C3 Elaboration: 5.1

C4 Parameters For Packages: N/A

D Human Subjects Including Annotators: No

D1 Instructions Given To Participants: N/A

D2 Recruitment And Payment: N/A

D3 Data Consent: N/A

D4 Ethics Review Board Approval: N/A

D5 Characteristics Of Annotators: N/A

E Ai Assistants In Research Or Writing: No

E1 Information About Use Of Ai Assistants: N/A

Author Submission Checklist: yes

Submission Number: 989