From Text to Trajectories: GPT-2 as an ODE Solver via In-Context

ACL ARR 2025 May Submission1648 Authors

Abstract: In-Context Learning (ICL) has emerged as a new paradigm in large language models (LLMs), enabling them to perform novel tasks by conditioning on a few examples embedded in the prompt. Yet, the highly nonlinear behavior of ICL for NLP tasks remains poorly understood. To shed light on its underlying mechanisms, this paper investigates whether LLMs can solve ordinary differential equations (ODEs) under the ICL setting. We formulate standard ODE problems and their solutions as sequential prompts and evaluate GPT-2 models on these tasks. Experiments on two types of ODEs show that GPT-2 can effectively learn a \textit{meta-ODE algorithm}, with convergence behavior comparable to, or better than, the Euler method, and achieve exponential accuracy gains with increasing numbers of demonstrations. Moreover, the model generalizes to out-of-distribution (OOD) problems, demonstrating robust extrapolation capabilities. These empirical findings provide new insights into the mechanisms of ICL in NLP and its potential for solving nonlinear numerical problems.

Paper Type: Short

Research Area: Interpretability and Analysis of Models for NLP

Research Area Keywords: In-Context Learning, GPT-2, ODE solver

Contribution Types: Model analysis & interpretability, Theory

Languages Studied: English

Submission Number: 1648