Unveiling Causal Calibration: How LLM Scale Influences Statistical Information Interpolation

Markus Englberger, Devendra Singh Dhami

Keywords: large language models, knowledge alignment, explicit and implicit knowledge

TL;DR: Alignment between explicit and implicit knoweldge in LLM is scale dependent.

Abstract: There are two ways an LLM can learn statistical and causal knowledge during training: either by explicit statements about the causal scenario in the training corpus (e.g., ``the probability of developing lung cancer is 20\% if one has smoked for at least 10 years'', ``X and Y are independent conditioned on Z'', ``X is a cause of Y'' etc.) or by implicit presentations of the same information via tabular data from surveys or scenario descriptions. In this paper, we probe whether the LLM internally unifies these two sources of information. To this end, we test whether fine-tuning LLMs on implicit presentations of statistical knowledge influences the LLM's answers when explicitly prompted for such knowledge. By creating in-context learning tasks where both explicit and implicit statistical information are present, we also test whether language models can unify these two modes during inference. Experiments suggest that larger language models have some shared method of storing these two sources of information, whereas smaller language models seem to be less capable of unifying these sources.

Submission Number: 40