MultiSessionCollab: Learning User Preferences with Memory to Improve Long-Term Collaboration

Shuhaib Mehri, Priyanka Kargupta, Tal August, Dilek Hakkani-Tür

Keywords: Multi-Session, Human-AI Collaboration, Memory, User Interaction

TL;DR: We present MultiSessionCollab, a benchmark for evaluating how well agent learn and adapt to user preferences to improve collaboration quality across sessions, and develop long-term collaborative agents equipped with memory to succeed in this setting.

Abstract: As conversational agents accumulate experience collaborating with users, adapting to user preferences is essential for fostering long-term relationships and improving collaboration quality over time. We introduce MultiSessionCollab, a benchmark that evaluates how well agents can learn user preferences and leverage them to improve collaboration quality throughout multiple sessions. To develop agents that succeed in this setting, we present long-term collaborative agents equipped with a memory that persists and refines user preference as interaction experience accumulates. Moreover, we demonstrate that learning signals can be derived from user simulator behavior in MultiSessionCollab to train agents to generate more comprehensive reflections and update their memory more effectively. Extensive experiments show that equipping agents with memory improves long-term collaboration, yielding higher task success rates, more efficient interactions, and reduced user effort. Finally, we conduct a human user study that demonstrates that memory helps improve user experience in real-world settings.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 77