APTBench: Benchmarking Agentic Potential of Base LLMs During Pre-Training

Jiarui Qin, Yunjia Xi, Junjie Huang, Renting Rui, di yin, Weiwen Liu, Yong Yu, Weinan Zhang, Xing Sun

Keywords: Agent Benchmark, Pre-Training Evaluation

TL;DR: Benchmark of pre-trained base LLMs (not post-trained models) for agentic potentials

Abstract:

With the rapid development of LLM-based agents, there is a growing trend to incorporate agent-specific data into the pre-training stage of LLMs, aiming to better align LLMs with real-world autonomous task execution.
However, current pre-training benchmarks primarily focus on isolated and static skills, e.g., common knowledge or mathematical/code reasoning, and fail to reflect model’s agentic capabilities. On the other hand, agent benchmarks are typically designed for post-trained models, requiring multi-turn task execution abilities that base models struggle to support. Thus, there is a compelling need for a benchmark that can evaluate agentic potentials during pre-training and guide the model training more effectively.
To address this gap, we propose APTBench, a framework that converts real-world agent tasks and successful trajectories into multiple-choice or text completion questions tailored for base models.
It focuses on core agentic abilities, e.g., planning and action, and covers key agent scenarios, software engineering and deep research.
Compared to existing general-purpose benchmarks, APTBench offers a more predictive signal of a model’s downstream performance as an agent, while remaining significantly more lightweight and cost-effective than full-scale, end-to-end agent evaluations after post-training.

Primary Area: datasets and benchmarks

Submission Number: 4408