Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMs

Itay Itzhak, Yonatan Belinkov, Gabriel Stanovsky

Keywords: bias, cognitive biases, large language models, LLM biases, bias analysis, instruction tuning, pretraining biases, causal experiments, bias evaluation, machine learning biases, model robustness, language model interpretability, bias sources

TL;DR: This work studies the origins of cognitive biases in LLMs using causal experiments and shows that while biases are slightly affected by instruction-tuning randomness, they primarily stem from pretraining.

Abstract:

Large language models (LLMs) exhibit cognitive biases -- systematic tendencies of irrational decision-making, similar to those seen in humans.
Prior work has found that these biases vary across models and can be amplified by instruction tuning.
However, it remains unclear if these differences in biases stem from pretraining, finetuning, or even random noise due to training stochasticity.
We propose a two-step causal experimental approach to disentangle these factors.
First, we finetune models multiple times using different random seeds to study how training randomness affects over $30$ cognitive biases.
Second, we introduce \emph{cross-tuning} -- swapping instruction datasets between models to isolate bias sources.
This swap uses datasets that led to different bias patterns, directly testing whether biases are dataset-dependent.
Our findings reveal that while training randomness introduces some variability, biases are mainly shaped by pretraining: models with the same pretrained backbone exhibit more similar bias patterns than those sharing only finetuning data.
These insights suggest that understanding biases in finetuned models requires considering their pretraining origins, especially given their high post-finetuning variability.
This perspective can guide future efforts to develop principled strategies for evaluating and mitigating bias in LLMs. See our code and models at: https://itay1itzhak.github.io/planted-in-pretraining.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html

Author Guide: I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html

Submission Number: 798