The Losing Winner: An LLM Agent That Predicts the Market but Loses Money

Youwon Jang, Joochan Kim, Byoung-Tak Zhang

Keywords: LLM, Reinforcement Learning, Finance, Generative AI

TL;DR: Fine-tuning an LLM for Bitcoin market state prediction improves accuracy but paradoxically worsens trading returns, exposing the dangers of proxy objectives and reward hacking in financial AI.

Abstract: Recent advancements in Large Language Models (LLMs) have spurred significant interest in their application to autonomous financial trading. This paper investigates the efficacy of fine-tuning a small-scale LLM, Qwen2.5-3B-Instruct, for Bitcoin (BTC) trading by framing the task as a market state prediction problem. Using daily price data, volume, and technical indicators, we train the model with Reinforcement Learning with Verifiable Reward (RLVR) to classify the next day's market into one of three states: bullish, consolidation, or bearish. Our experiments reveal a critical paradox: the fine-tuned agent achieves significantly higher classification accuracy compared to a zero-shot baseline, demonstrating a clear ability to learn the defined task. However, when deployed in a simulated trading environment, this improved predictive power results in a lower cumulative return than the baseline. We attribute this divergence to a classic case of objective mismatch and reward hacking. The agent optimizes for the simple, discrete reward of a correct classification, a proxy objective that fails to capture the complexities of profitable trading, such as risk management and the magnitude of price movements. This work serves as a cautionary case study, highlighting the inherent challenges of designing effective reward functions for LLM-based financial agents and underscoring the importance of aligning proxy tasks with the ultimate goal of profit maximization.

Submission Number: 69