Understanding LoRA As Knowledge Memory: An Empirical Analysis

Seungju Back, Dongwoo Lee, Naun Kang, Taehee Lee, S. K. Hong, Youngjune Gwon, Sungjin Ahn

Keywords: Large Language Model, Low Rank Adaptation, LLM Memory

Abstract: Continuous knowledge updating in Large Language Models (LLMs) is a critical challenge. While existing methods like Retrieval-Augmented Generation (RAG) and In-Context Learning (ICL) offer solutions, they are constrained by retrieval quality and context length. Departing from the conventional view of LLM memory that relies on context, this work highlights a novel parametric approach via Low-Rank Adaptation (LoRA). Although a few studies have hinted at this potential, LoRA's mechanics and optimal usage as a memory component remain largely unexplored. To bridge this gap, we conduct the first systematic and comprehensive empirical study of LoRA-based knowledge memory. Our analysis spans multiple dimensions, including the fundamental memory characteristics of LoRA, how to optimize a single LoRA, the possibilities of combining multiple LoRAs, and its synergy with existing methods in complex scenarios. Ultimately, this paper presents the first systematic framework for LoRA-based memory, offering foundational insights and actionable guidelines to future research and application.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 24020