Language Models Linearly Represent Sentiment

Oskar John Hollinsworth, Curt Tigges, Atticus Geiger, Neel Nanda

Track: Full paper

Keywords: Mechanistic Interpretability, Linear Representation Hypothesis, Causal Analysis, Large Language Models, Sentiment

TL;DR: Sentiment is represented by a single causal direction in LLMs and is summarized at placeholder tokens

Abstract:

Sentiment is a pervasive feature in natural language text, yet it is an open question how sentiment is represented within Large Language Models (LLMs). In this study, we reveal that across a range of models, sentiment is represented linearly: a single direction in activation space mostly captures the feature across a range of tasks with one extreme for positive and the other for negative. In a causal analysis, we isolate this direction using interventions and show it is causal in both toy tasks and real world datasets such as Stanford Sentiment Treebank.

We analyze the mechanisms that involve this direction and discover a phenomenon which we term the summarization motif: sentiment is not just represented on valenced words, but is also summarized at intermediate positions without inherent sentiment, such as punctuation and names. We show that in SST classification, ablating the sentiment direction across all tokens results in a drop in accuracy from 100\% to 62\% (vs. 50\% random baseline), while ablating the summarized sentiment direction at comma positions alone produces close to half this result (reducing accuracy to 82\%).

Submission Number: 17