Reflection System for the Abstraction and Reasoning Corpus
Keywords: reflection systems, language models with reasoning capabilities, abstraction and reasoning corpus
Abstract: The Abstraction and Reasoning Corpus (ARC) benchmarks broad generalization in artificial intelligence, and presents a significant challenge to existing machine learning models and program synthesis solvers. In this work, we introduce a Reflection System for ARC. It combines Large Language Models (LLMs) and a program synthesis solver based on a Domain Specific Language (DSL). We analyse the accuracy of LLMs on ARC and demonstrate unsatisfactory results. We create AugARC, an augmented ARC benchmark, which consistently improves the performance of LLMs compared to the normal ARC benchmark. Using augmented ARC data, we fine-tune LLMs and observe a significant gain in ARC accuracy after training. By utilizing reflection, we combine LLMs and a previous DSL solver into our Reflection System for abstraction and reasoning. Our approach outperforms the previous publicly available ARC systems that consist solely of LLMs or DSL solvers. The proposed Reflection System motivates research to advance previous ARC attempts by combining the advantages of LLMs and program synthesis solvers with reflection.
Submission Number: 5
Loading