A Simple Loss Function for Convergent Algorithm Synthesis using RNNs

Alexandre Salle, Shervin Malmasi

Keywords: Recurrent Neural Networks, logical extrapolation, algorithm synthesis, iterative reasoning

TL;DR: We show the effectiveness of an alternative loss function for convergent algorithm synthesis using RNNs which, when compared to the state-of-the-art, has a simpler formulation, increased training efficiency, and greater robustness.

Abstract: Running a Recurrent Neural Network (RNN) over the same input multiple times, or iterative reasoning, enables logical extrapolation, where a model can be run on problems larger than the models were trained on. The loss function used to train these networks has a profound impact on their extrapolation ability. In this paper, we propose using a simple loss function called the Delta Loss (Salle & Prates, 2019). We show that the Delta Loss, like the state-of-the-art Progressive Loss (Bansal et al., 2022), leads to convergent algorithm synthesis, but with a simpler formulation, increased training efficiency, and greater robustness.