It's all in the heads: An investigation of domain knowledge infusion into LLMs

Nikita Borodin, Maria Krylova, Artem Zabolotnyi, Dmitry Aspisov, Egor Shikov, Nikita Tyuplyaev, Oleg Travkin, Roman Alferov, Dmitry Vinichenko

Keywords: Large Language Models, Continued Pre-Training, Singular Value Decomposition, Linear Mode Connectivity, Domain Knowledge

Abstract:

While large language models (LLMs) are widely studied, the mechanisms by which they internalize knowledge from specialized domains remain poorly understood. To investigate this, we analyze the continual pre-training (CPT) paradigm, where a base model is further pre-trained on a curated, domain-specific corpus. Through a study on diverse domains, including mathematics, instruction, code, and text data, we uncover novel properties of this process. By analyzing SVD decompositions of model weights we determine that the difference before and after CPT can be attributed predominantly to changes in singular vectors. We identify **head heterogeneity** in the behavior of attention weight matrices. We investigate the effect of rewinding attention heads on model quality by ordering them according to various scalar criteria. Based on our analysis we propose a novel head importance criterion which allows to either truncate up to **60**% heads in the model increment or to achieve up to **4**% quality increase upon partial head rewinding to the pre-train state. Further, we discover **domain connectivity** — *i.e.*, the ability to linearly interpolate between CPT checkpoints on different domains without significant quality loss, and discuss key quality drivers of this phenomenon. To foster further research, we provide an open-source scalable toolkit for performing spectral analysis on models with billions of parameters - NetInspect. The code is
available at https://anonymous.4open.science/r/netinspect-EF67

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 13125