Vector Memory and Role-Conditioned Multi-Agent Systems: Two Extensions to Improve Reflexion for Language Model Self-Improvement

TMLR Paper8541 Authors

Abstract:

REFLEXION improves language model performance through verbal self-reflection, but two design choices limit its reach. Its memory
is a recency-ordered sliding window that evicts old reflections as new ones arrive, regardless of which are actually relevant. Moreover, a single model simultaneously generates a solution, critiques it, and plans the next attempt, roles that genuinely benefit from separation. In this paper, we address both of these limitations. We replace the sliding window with vector episodic memory, which stores Sentence-BERT embeddings alongside each reflection and retrieves them based on cosine similarity rather than recency. We also split the single agent into a Generator, a Critic, and a Verifier, each of which draws on a shared role-conditioned memory pool. The results are quite encouraging. In a controlled benchmark in which 9 distractor tasks bury the relevant memories, temporal memory fails (0% recall). In contrast, vector memory succeeds without exception (100%, zero variance across 3 trials), at a constant ~14 ms overhead that stays flat up to 50,000 stored reflections. On 164 HumanEval coding tasks with Google Gemini~2.5 Flash, the vector extension reaches Pass@3= 92.7% (+3.7~pp, p=0.033, d=0.127), and the multi-agent system reaches Pass@3= 96.3% (+7.9~pp, p<0.001, d=0.301) with Pass@1= 93.9% (+12.2~pp over the modular baseline), a first-attempt gain that predates any reflection at all. Our results suggest that semantic retrieval is particularly important when tasks are correlated across sessions, while role separation provides the greatest benefit on independent code-generation tasks.

Submission Type: Long submission (more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=SQmsFm7b1j

Changes Since Last Submission: The Introduction section has been substantially revised to clearly articulate the background and motivation, identify the research gap, develop the underlying theoretical intuition, and present the proposed approach along with its main contributions.

Assigned Action Editor: ~Marc_Lanctot1

Submission Number: 8541