Low-Rank Embedding Adaptation for Models with Expanding Vocabularies

Keywords: low-rank adaptation, LoRA, embeddings, vocabulary expansion, incremental learning, sequential recommendation, parameter efficiency

TL;DR: Population-specific low-rank subspaces extend LoRA from weight matrices to embedding tables, resolving the base-vs-new quality tradeoff in vocabulary expansion across recommendation and LLMs.

Abstract: Pretrained models are routinely extended with new vocabulary entries, for example new items in recommenders and new tokens in LLMs. We show that jointly training old and new embeddings has a hidden failure mode: old-entry quality degrades while new entries are still learning. On sequential recommendation, old items overfit while new items improve, forcing premature early-stopping; on LLMs, base-token perplexity rises within 1–2 epochs. We propose *population-specific low-rank subspaces* that extend the low-rank adaptation principle from weight matrices to embedding tables, with separate shared projections for base and new entries. On sequential recommendation (SASRec, GRU4Rec; Taobao, MerRec), our methods Pareto-dominate joint fine-tuning and continual-learning baselines. On LLM vocabulary expansion (GPT-2, Pythia-410M, Pythia-1.4B; BillSum, PubMed), our low-rank methods improve overall perplexity over joint training in 8 out of 9 model-scale cells, outperforming full-rank frozen baselines in 8/9 cells at 2× compression and 6/9 at 4× compression.

Submission Number: 102