Improving Multi-step RAG with Hypergraph-based Memory

ICLR 2026 Conference Submission16946 Authors

Keywords: retrieval augmented generation, memory, hypergraph

TL;DR: We propose a hypergraph-based memory mechanism that enables the evolving of memory to form high-order correlation from stored information, improving LLMs to better interact with external datasources for resolving long-context tasks.

Abstract:

Multi-step retrieval-augmented generation (RAG)
has become a widely adopted strategy for enhancing large language models (LLMs) on tasks that demand global comprehension and intensive reasoning.
Many RAG systems incorporate a working memory module to consolidate retrieved information. However, existing memory designs function primarily as passive storage that accumulates isolated facts for the purpose of condensing the lengthy inputs and generating new sub-queries through deduction.
This static nature overlooks the crucial high-order correlations among primitive facts, the compositions of which can often provide stronger guidance for subsequent steps.
Therefore, their representational strength and impact on multi-step reasoning and knowledge evolution are limited; resulting in fragmented reasoning and weak global sense-making capacity in extended contexts. We introduce \textsc{HGMem}, a hypergraph-based memory mechanism that extends the concept of memory beyond simple storage into a dynamic, expressive structure for complex reasoning and global understanding.
In our approach, memory is represented as a hypergraph whose hyperedges correspond to distinct memory units, enabling the progressive formation of higher-order interactions within memory. This mechanism connects facts and thoughts around the focal problem, evolving into an integrated and situated knowledge structure that provides strong propositions for deeper reasoning in subsequent steps. We evaluate \textsc{HGMem} on several challenging datasets designed for global sense-making. Extensive experiments and in-depth analyses show that our method consistently improves multi-step RAG and substantially outperforms strong baseline systems across diverse tasks.

Primary Area: generative models

Submission Number: 16946