Parallel Algorithms Align with Neural Execution

Valerie Engelmayer, Dobrik Georgiev Georgiev, Petar Veličković

Keywords: Parallel Algorithms, Neural Algorithmic Reasoning, Graph Neural Networks

TL;DR: Teaching NN to execute parallel algorithms instead of sequential ones drastically reduces training times and often improves OOD performance by a lot as well.

Abstract: Neural algorithmic reasoners are parallel processors. Teaching them sequential algorithms contradicts this nature, rendering a significant share of their computations redundant. Parallel algorithms however may exploit their full computational power, therefore requiring fewer layers to be executed. This drastically reduces training times, as we observe when comparing parallel implementations of searching, sorting and finding strongly connected components to their sequential counterparts on the CLRS framework. Additionally, parallel versions achieve strongly superior predictive performance in most cases.

Submission Type: Full paper proceedings track submission (max 9 main pages).

Submission Number: 40