[Re] End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinking

Sean Michael McLeish, Long Tran-Thanh

Keywords: rescience c, Algorithmic Reasoning, Deep Thinking, Algorithm Synthesis, Python, machine learning, Deep Learning

TL;DR: Reproducibility Study of "End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinking."

Abstract:

Scope of Reproducibility:
In this report, we aim to validate the claims of Bansal et al. These are that the recurrent architecture presented, with skip connections and a progressive loss function, prevent the original problem being forgotten or corrupted during processing allowing for the recurrent module to be applied indefinitely and that this architecture avoids the overthinking trap. We use both code released by the authors and newly developed to recreate many results presented in the paper. Additionally, we present analysis of the newly introduced alpha hyperparameter and investigate interesting perturbation behaviour of prefix sums models. Further, we conduct a hyperparameter search and provide an analysis of the Asymptotic Alignment scores of the models presented.

Methodology:
We use the PyTorch code released by the authors to replicate accuracy experiments. We then, independently, develop our own PyTorchFI to replicate perturbation experiments presented by Bansal et al. Overall, providing a replication of all results shown in the main body of the paper. We then extend these results, providing an analysis of the alpha hyperparameter, analysis of perturbation recovery, Asymptotic Alignment scores and a hyperparameter search. We used both a Nvidia RTX 2080Ti GPU and sets of three NVIDIA Quadro RTX6000 GPUs, taking a total of 982.2 GPU hours to create all results presented in the main body of this report.

Results:
We verify the authors' claims by replicating the experiments presented by Bansal et al. All of our experiments show identical results to the ones presented in the paper, apart from perturbation testing for which we provide an additional in depth analysis. We also provide an analysis of the new alpha hyperparameter and a hyperparameter search.

What was easy:
The code provided by Bansal et al gave clear instructions on how to use it, along with pretrained models being available for all problems.

What was difficult:
Chess models required a considerable amount of time to train, putting a drain on resources. Also, code for reproducing perturbation results was not available so this had to developed from scratch.

Communication with original authors:
We had good communication with the original authors, both emailing and meeting online.

Paper Url: https://arxiv.org/pdf/2202.05826.pdf

Paper Review Url: https://openreview.net/forum?id=PPjSKy40XUB

Paper Venue: NeurIPS 2022

Confirmation: The report pdf is generated from the provided camera ready Google Colab script, The report metadata is verified from the camera ready Google Colab script, The report contains correct author information., The report contains link to code and SWH metadata., The report follows the ReScience latex style guides as in the Reproducibility Report Template (https://paperswithcode.com/rc2022/registration)., The report contains the Reproducibility Summary in the first page., The latex .zip file is verified from the camera ready Google Colab script

Journal: ReScience Volume 9 Issue 2 Article 3

Doi: https://www.doi.org/10.5281/zenodo.8173654

Code: https://archive.softwareheritage.org/swh:1:dir:be1a763d08c1af34a2721fad6b9bfe8c99527220