When Valid Signals Fail: Regime Boundaries Between LLM Features and RL Trading Policies

Zhengzhe Yang

Keywords: LLM, PPO, Finance

TL;DR: LLM feature extraction for RL trading agents

Abstract:

Can large language models (LLMs) generate continuous
numerical features that improve reinforcement
learning (RL) trading agents? We build
a modular pipeline where a frozen LLM serves
as a stateless feature extractor, transforming unstructured
daily news and filings into a fixeddimensional
vector consumed by a downstream
PPO agent. We introduce an automated promptoptimization
loop that treats the extraction prompt
as a discrete hyperparameter and tunes it directly
against the Information Coefficient—the Spearman
rank correlation between predicted and realized
returns—rather than NLP losses. The optimized
prompt discovers genuinely predictive features
(IC above 0.15 on held-out data). However,
these valid intermediate representations do not
automatically translate into downstream task performance:
during a distribution shift caused by a
macroeconomic shock, LLM-derived features add
noise, and the augmented agent under-performs
a price-only baseline. In a calmer test regime
the agent recovers, yet macroeconomic state variables
remain the most robust driver of policy improvement.
Our findings highlight a gap between
feature-level validity and policy-level robustness
that parallels known challenges in transfer learning
under distribution shift.

Submission Number: 7