Forecasting With LLMs: Improved Generalization Through Feature Steering

Humzah Merchant, Bradford Levy

Keywords: Forecasting, Prediction Markets

TL;DR: Amplifying features associated with time-awareness in LLMs can reduce reliance on information from after a cutoff specified at inference-time

Abstract: Successful forecasting involves identifying patterns between historical and future states of the world which generalize to future observations. We apply LLMs to a variety of forecasting tasks and inspect their internal states using sparse autoencoders to understand whether they appear to rely on time-specific pieces of knowledge versus generalizable patterns. Our analyses identify features associated with both time-aware reasoning and look-ahead-biased reasoning. We then apply the LLMs to an ***entirely different domain*** and intervene on these features. We find that amplifying time-awareness features ***substantially*** reduces look-ahead bias on forecasting prompts while preserving general reasoning performance. In contrast, steering the candidate look-ahead-bias features does not produce an effect. These results suggest that interpretable temporal features can be used to causally shift LLMs toward more historically grounded reasoning.

Submission Number: 91