UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Language Models

Xiaojie Gu, Ziying Huang, Jia-Chen Gu, Kai Zhang

Keywords: Model Editing, Lifelong Learning, Large Language Models

TL;DR: We propose UltraEdit, a scalable method for lifelong editing in LLMs that supports over 2M efficient edits with minimal compute and SOTA results. We also introduce UltraEditBench, the largest benchmark for model editing to date.

Abstract:

Lifelong learning enables large language models (LLMs) to adapt to evolving information by continually updating their internal knowledge. An ideal system should support efficient, wide-ranging updates while preserving existing capabilities and ensuring reliable deployment.
Model editing stands out as a promising solution for this goal, offering a focused and efficient way to revise a model’s internal knowledge. Although recent paradigms have made notable progress,
they often struggle to meet the demands of practical lifelong adaptation at scale.
To bridge this gap, we propose UltraEdit, a *training-*, *subject-*, and *memory-free* approach that is well-suited for ultra-scalable, real-world lifelong model editing.
UltraEdit fundamentally differs from traditional paradigms by computing parameter shifts in one step using only a hidden state and its gradient, making the approach simple yet efficient.
To improve scalability in lifelong settings, UltraEdit employs a *lifelong normalization* strategy that continuously updates feature statistics across turns, allowing it to adapt to distributional shifts and maintain consistency over time.
UltraEdit achieves editing speeds over **7× faster** than the previous state-of-the-art method, which was also the fastest known approach, while using less than **1/4 the VRAM**. This makes it the **only** method currently capable of editing a 7B LLM on a 24GB consumer-grade GPU.
Furthermore, we construct UltraEditBench, the largest dataset in the field to date with over **2M** editing pairs, and demonstrate that our method supports up to **2M** edits while maintaining high accuracy.
Comprehensive experiments on five datasets and six models show that UltraEdit consistently achieves superior performance across diverse model editing scenarios, taking a further step towards safe and scalable lifelong learning.
We will release the code and dataset upon acceptance.

Primary Area: generative models

Submission Number: 5877