Low-rank embedding adaptation for models with expanding vocabularies

Pretrained models are routinely extended with new vocabulary entries, for example new items in recommender systems and new tokens in large language models (LLMs). We show that jointly training old and new embeddings has a hidden failure mode: old-entry quality degrades while new entries are still learning. On sequential recommendation, old items overfit while new items improve, forcing premature early stopping; on LLMs, base-token perplexity rises within 1–2 epochs of joint training. We propose population-specific low-rank subspaces that extend the low-rank adaptation principle to embedding tables: each entry is parameterized as ei = ¯ei + uiV>k, where Vk is shared within a population and separate across populations with different data regimes. The shared Vk provides implicit regularization for sparse new entries while the separation prevents gradient interference between populations. Three instantiations (Freeze-SV, Freeze1-SV, Dual-SV) cover different regimes. On sequential recommendation (SASRec and GRU4Rec architectures; Taobao and MerRec datasets), our methods Pareto-dominate joint fine-tuning and continual-learning baselines. On LLM vocabulary expansion (GPT-2, Pythia-410M, Pythia-1.4B on BillSum and PubMed), freezing base embeddings (the core of our approach) improves overall perplexity over joint training in 8 of 9 model-scale cells, with the low-rank parameterization providing 2–4× compression at matched quality.