PBEBench: A Multi-Step Programming by Examples Reasoning Benchmark inspired by Historical Linguistics

ICLR 2026 Conference Submission5217 Authors

Keywords: Dynamic Benchmark Generation, LLM Reasoning, Programming by Examples, Code Generation, Historical Linguistics

Abstract:

Although many benchmarks evaluate the reasoning abilities of Large Language Models (LLMs) within domains such as mathematics, coding, or data wrangling, few abstract away from domain specifics to examine reasoning as a capability in and of itself.
We contribute a novel type of benchmark evaluating the inductive reasoning capabilities of LLMs that is inspired by the forward reconstruction task from historical linguistics but is formulated in an extremely simple, general way (in the form of Programming by Examples or PBE).
The task involves generating a cascade of simple string rewrite programs to transform a given list of input strings into a list of desired output strings.
We present a fully automated pipeline that programmatically generates problems of this type with controllable difficulty (varying, for example, ground truth cascade lengths), enabling scalable evaluation of reasoning models while avoiding contamination.
Using this approach, we construct two benchmarks: PBEBench-Lite, which efficiently stratifies models of varying capabilities, and PBEBench, which requires models to induce programs more similar (in complexity) to those constructed by historical linguists.
Our experiments reveal a substantial performance gap between models that leverage test-time compute or LCoT (long chain-of-thought) reasoning and those that do not.
Moreover, although recent models such as gpt-oss-120b and GPT-5 show promise, the solve rate for both of them drops below 5% for hard instances of the PBEBench dataset (ground truth cascade lengths of 20 and 30, respectively), falling well short of realistic historical linguistics requirements even with computationally expensive, popular scaling techniques from the PBE and reasoning literature.
Additionally, we also study the effectiveness of different scaling strategies and the impact of various hyperparameters on the difficulty of the generated data using gpt-oss-120b, the best-performing open-source model. We plan to open-source our code and benchmark snapshots to enable reproducibility and future work.

Primary Area: datasets and benchmarks

Submission Number: 5217