Go to ACL ARR 2026 January homepageThe Impact of Off-Policy Training Data on Probe Generalisation
Keywords: Large language models, Mechanistic interpretability, Linear probes, Moderation, AI safety, AI control, Misalignment, Sycophancy
Abstract: Probing has emerged as a promising method for monitoring large language models (LLMs), enabling cheap inference-time detection of concerning behaviours. However, natural examples of many behaviours are rare, forcing researchers to rely on synthetic or off-policy LLM responses for training probes. We systematically evaluate how off-policy data influences probe generalisation across eight distinct LLM behaviours. Testing linear and attention probes across multiple LLMs, we find that training data generation strategy can significantly affect probe performance, though the magnitude varies greatly by behaviour. The largest generalisation failures arise for behaviours defined by response ``intent'' (e.g. strategic deception) rather than text-level content (e.g. usage of lists). We then propose a useful test for predicting generalisation failures in cases where on-policy test data is unavailable: successful generalisation to incentivised data (where the model was coerced) strongly correlates with high performance against on-policy examples. Based on these results, we predict that current deception probes may fail to generalise to real monitoring scenarios. Additionally, our finding that off-policy data can yield more reliable probes than on-policy data from a sufficiently different setting underscores the need for new monitoring methods that better handle all types of distribution shift.
Paper Type: Long
Research Area: Safety and Alignment in LLMs
Research Area Keywords: Language Modeling, Ethics, Bias, and Fairness, Interpretability and Analysis of Models for NLP, Semantics: Lexical and Sentence-Level, Question Answering
Contribution Types: Model analysis & interpretability, NLP engineering experiment, Reproduction study
Languages Studied: English
Submission Number: 3832
Loading