ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm Engineering

Yuki Imajuku, Kohki Horie, Yoichi Iwata, Kensho Aoki, Naohiro Takahashi, Takuya Akiba

Keywords: benchmark, evaluations, dataset, tasks, engineering, coding, algorithm, heuristic, optimization, language, large language models, multimodal, vision, agents, scaffold, swe, mle, ale

TL;DR: We introduce ALE-bench, a new benchmark for evaluating AI systems on score-based algorithmic programming contests.

Abstract:

How well do AI systems perform in algorithm engineering for hard optimization problems in domains such as package-delivery routing, crew scheduling, factory production planning, and power-grid balancing?
We introduce $\textit{ALE-Bench}$, a new benchmark for evaluating AI systems on score-based algorithmic programming contests. Drawing on real tasks from the AtCoder Heuristic Contests, ALE-Bench presents optimization problems that are computationally hard and admit no known exact solution.
Unlike short-duration, pass/fail coding benchmarks, ALE-Bench encourages iterative solution refinement over long time horizons.
Our software framework supports interactive agent architectures that leverage test-run feedback and visualizations.
Our evaluation of frontier LLMs revealed that while they demonstrate high performance on specific problems, a notable gap remains compared to humans in terms of consistency across problems and long-horizon problem-solving capabilities. This highlights the need for this benchmark to foster future AI advancements.

Dataset URL: https://huggingface.co/datasets/SakanaAI/ALE-Bench

Code URL: https://github.com/SakanaAI/ALE-Bench

Primary Area: Datasets & Benchmarks for applications in language modeling and vision language modeling

Submission Number: 270