Position: Bitter lesson of the ARC-AGI Challenge: Intelligence may look very different in machines and humans

Soumya Banerjee, Andrew Krapivin

TL;DR: Machines may excel at solving benchmarks like ARC, but their reasoning and intelligence may differ fundamentally from human approaches, prompting the need to rethink terms like reasoning

Abstract:

The Abstraction and Reasoning Corpus (ARC) and the associated ARC-AGI challenge, serve as a benchmark to evaluate a system's ability to demonstrate core reasoning skills and abstraction. Recently, the newest, as-of-yet unreleased model by OpenAI, o3, passed this benchmark.

These breakthroughs have prompted discussion on whether current AI techniques are advancing genuine reasoning capabilities or engage in merely pattern-matching behaviours. The "Bitter Lesson" articulated by Rich Sutton argues that general methods with large computational resources tend to outperform specialised approaches. This insight also applies to ARC, as many winning approaches rely heavily on data augmentation and test-time training, raising philosophical and methodological questions.

We explore the tension between the goals of ARC and the methods that are applied and whether the need for new concepts and neologisms could better describe machine reasoning. Terms like "reasoning" and "understanding" may need redefinition to account for the unique characteristics of machine intelligence.

This position paper argues that reasoning and intelligence may be very different in machines and humans. Recognising that intelligence is task-specific and comes in various forms may help us appreciate that animals, humans and machines have different kinds of intelligence, and help us design better benchmarks for machines.

Primary Area: Other topic (use sparingly and specify relevant keywords)

Keywords: abstraction and reasoning, reasoning in machines, reasoning in humans

Submission Number: 147