PerLTQA: A Personal Long-Term Memory Dataset for Memory Classification, Retrieval, and Synthesis in Question Answering

Anonymous

Abstract: Long-term memory plays a critical role in personal interaction, considering long-term memory can better leverage world knowledge, historical information, and preferences in dialogues. Our research introduces PerLTQA, an innovative QA dataset that combines semantic and episodic memories, including world knowledge, profiles, social relationships, events, and dialogues. This dataset is collected to investigate the use of personalized memories, focusing on social interactions and events in the QA task. PerLTQA features two types of memory and a comprehensive benchmark of 8,593 questions for 30 characters, facilitating the exploration and application of personalized memories in Large Language Models (LLMs). Based on PerLTQA, we propose a novel framework for memory integration and generation, consisting of three main components: memory classification, memory retrieval, and memory synthesis. We evaluate this framework using five LLMs and three retrievers. Experimental results demonstrate that BERT-based classification models significantly outperform LLMs such as ChatGLM3 and ChatGPT in the memory classification task. Furthermore, our study highlights the importance of effective memory integration in the QA task.

Paper Type: long

Research Area: Resources and Evaluation

Contribution Types: NLP engineering experiment, Data resources

Languages Studied: English, Chinese

Preprint Status: We plan to release a non-anonymous preprint in the next two months (i.e., during the reviewing process).

A1: yes

A1 Elaboration For Yes Or No: Sec Limitation

A2: yes

A2 Elaboration For Yes Or No: Sec Limitation

A3: yes

A3 Elaboration For Yes Or No: Sec Abstract

B: yes

B1 Elaboration For Yes Or No: Sec 3

B2: yes

B2 Elaboration For Yes Or No: Sec Ethics Statement

B3: yes

B3 Elaboration For Yes Or No: Sec 4; Sec 5

B4: yes

B4 Elaboration For Yes Or No: Sec 3; Sec Appendix

B5: yes

B5 Elaboration For Yes Or No: Sec 3

B6: yes

B6 Elaboration For Yes Or No: Sec 3; Sec Appendix

C: yes

C1: yes

C1 Elaboration For Yes Or No: Sec 4;

C2: yes

C2 Elaboration For Yes Or No: Sec 4; Sec Appendix

C3: yes

C3 Elaboration For Yes Or No: Sec 4; Sec 5

C4: yes

C4 Elaboration For Yes Or No: Sec 4

D: yes

D1: yes

D1 Elaboration For Yes Or No: Sec 3; Sec Appendix

D2: yes

D2 Elaboration For Yes Or No: Sec Ethics Statement

D3: yes

D3 Elaboration For Yes Or No: Sec 3

D4: yes

D4 Elaboration For Yes Or No: Sec Ethics Statement

D5: yes

D5 Elaboration For Yes Or No: Sec Ethics Statement

E: yes

E1: yes

E1 Elaboration For Yes Or No: Sec 3