Exploring the Memory Ability of Large Language Models
Abstract: The memory ability is critical for large language models (LLMs).
However, the difference between small LLMs and large LLMs in memory ability is still blurring.
In this paper, we study the memory ability of them.
To this end,
we create a knowledge based dataset with frequency.
This dataset is based on Wikidata5M,
and we count the number of co-occur documents in Wikipedia and Baidu Baike of head and tail entity as the fact's frequency.
Our experiments demonstrate that large LLMs has strong memory ability.
They can remember most facts even with low frequency.
Small LLMs, on the contrary, can only remember part of the high frequency facts, not to mention low frequency facts.
Paper Type: Short
Research Area: Language Modeling
Research Area Keywords: pre-training,corpus creation
Contribution Types: NLP engineering experiment, Data resources
Languages Studied: Chinese
Submission Number: 2468
Loading