Learning to Learn-at-Test-Time: Language Agents with Learnable Adaptation Policies

Zhanzhi Lou, Hui Chen, Yibo Li, Qian Wang, Bryan Hooi

Keywords: test-time learning, language agents, meta-learning, adapting agents, evolutionary search

TL;DR: We meta-learn how a language agent adapts across episodes at test time, replacing hand-crafted adaptation heuristics with an evolved natural-language meta-prompt that transfers zero-shot to unseen tasks.

Abstract: Test-Time Learning (TTL) enables language agents to iteratively refine their performance through repeated interactions with the environment at inference time. At the core of TTL is an adaptation policy that updates the actor policy based on experience from previous episodes, thereby improving future behavior. Existing methods typically leave this policy to LLMs' emergent abilities rather than optimizing it explicitly. We argue that adaptation policies should be learned from task environments, not hand-engineered from human intuition. To achieve this, we introduce Meta-TTL, a framework that formulates the discovery of effective adaptation policies as a bi-level optimization problem. Within this framework, the inner loop executes the standard TTL process, measuring how effectively a candidate learning policy, instantiated as a natural-language meta-prompt, helps an agent correct errors across sequential episodes. Using the agent performance as a guiding signal, the outer loop employs evolutionary search over a diverse distribution of training tasks to iteratively refine the adaptation policy. We evaluate Meta-TTL on Jericho and WebArena-Lite across both in-distribution (ID) and out-of-distribution (OOD) settings, using multiple meta-agent backbones. Results on both benchmarks show that Meta-TTL consistently outperforms hand-crafted baselines, suggesting that the optimized adaptation policy encodes transferable strategies that generalize beyond the training task distribution.

Track: Regular Paper (9 pages)

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 312