AssoMem: Scalable Memory QA with Multi-Signal Associative Retrieval

ICLR 2026 Conference Submission12791 Authors

Keywords: memory-augmented LLM, scalable retrieval, memory question answering

TL;DR: We propose AssoMem, a novel framework that solves the challenge of accurate, scalable memory QA by forming an associative memory graph and adaptively fusing multi-dimensional retrieval signals, resulting in state-of-the-art performances.

Abstract: Accurate recall from large-scale memories remains a core challenge for memory-augmented AI assistants performing question answering (QA), especially in similarity-dense scenarios where existing methods mainly rely on semantic distance to the query for retrieval. Inspired by how humans link information associatively, we propose AssoMem, a novel framework constructing an associative memory graph that anchors dialogue utterances to automatically extracted clues. This structure provides a rich organizational view of the conversational context and facilitates importance-aware ranking. Further, AssoMem integrates multi-dimensional retrieval signals—relevance, importance, and temporal alignment—using an adaptive mutual information (MI)-driven fusion strategy. Extensive experiments across three benchmarks and a newly introduced dataset, MeetingQA, demonstrate that AssoMem consistently outperforms state-of-the-art baselines, verifying its superiority in context-aware memory recall.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 12791