Memory Tokens: Large Language Models can generate reversible Sentence Embeddings

ACL ARR 2025 February Submission4344 Authors

Abstract: In this work, we observe an interesting phenomenon: it is possible to generate reversible sentence embeddings that allow an LLM to reconstruct the original text exactly, without modifying the model's weights. This is achieved by introducing a special memory token, whose embedding is optimized through training on a fixed sequence. When prompted with this embedding, the model reconstructs the fixed sequence exactly. We evaluate this phenomenon across different datasets, sequence lengths, and model scales. Notably, Llama 3.1 8B successfully reconstructs all tested sequences. Our findings highlight an interesting capability of LLMs and suggest potential applications in memory-based retrieval, compression, and controlled text generation.

Paper Type: Short

Research Area: Language Modeling

Research Area Keywords: prompting, fine-tuning, scaling, retrieval-augmented models

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: english, spanish

Submission Number: 4344