Towards \textit{Effective Theory} of LLMs: A Representation Learning Approach

Muhammed Ustaomeroglu, Guannan Qu

Keywords: Effective Theory, Interpretability, Representation learning

TL;DR: RET learns high-level LLM “macrostates” that make reasoning easier to interpret, predict, and steer.

Abstract:

We propose Representational Effective Theory (RET), a framework for describing large language model computation in terms of learned macrostates rather than microscopic activation details.
RET learns these macrostates from hidden-state trajectories using a BYOL/JEPA-style self-supervised objective, coarse-graining activations into macrovariables that preserve higher-level structure relevant for prediction and interpretation.
We evaluate whether these macrovariables are practically relevant for interpretability: RET yields temporally consistent states that reveal ``mental-state'' trajectories of reasoning, capture high-level semantic structure, support early prediction of behavioral outcomes such as sycophancy, and provide causal handles for steering generations toward interpretable computational phases.
Together, these results suggest that LLM computation admits useful effective descriptions via RET: high-level, dynamically meaningful variables that support interpretation, prediction, and intervention.

Submission Number: 180