Algorithm Development in Neural Networks: Insights from the Streaming Parity Task

Loek van Rossem, Andrew M Saxe

TL;DR: We explain in a simple setting how out-of-distribution generalization can occur.

Abstract: Even when massively overparameterized, deep neural networks show a remarkable ability to generalize. Research on this phenomenon has focused on generalization within distribution, via smooth interpolation. Yet in some settings neural networks also learn to extrapolate to data far beyond the bounds of the original training set, sometimes even allowing for infinite generalization, implying that an algorithm capable of solving the task has been learned. Here we undertake a case study of the learning dynamics of recurrent neural networks trained on the streaming parity task in order to develop an effective theory of algorithm development. The streaming parity task is a simple but nonlinear task defined on sequences up to arbitrary length. We show that, with sufficient finite training experience, RNNs exhibit a phase transition to perfect infinite generalization. Using an effective theory for the representational dynamics, we find an implicit representational merger effect which can be interpreted as the construction of a finite automaton that reproduces the task. Overall, our results disclose one mechanism by which neural networks can generalize infinitely from finite training experience.

Lay Summary:

A child in primary school math class might be told that 3+7=10. The next day when asked what 3+7 equals, they can provide the correct answer, but cannot for instance say what 4+5 equals, since they have not seen this before. One day, after seeing enough examples, something clicks, and now they are now able to give the correct sum for any pair of numbers. When we train neural networks, sometimes we see that a similar "click" happens.

In this paper, we explain this phenomenon for a simple computational task. We argue that the quickest way for a neural network to learn to reproduce examples is by learning an internal representation of these examples, which ignores details irrelevant to solving the task. This simpler representation will start to resemble the original task, and as a result the network will be able to compute previously unseen examples.

Understanding why such a "click" can occur in a neural network might help understand why something similar happens in the brain. Additionally, knowing when neural networks really learn to compute is important, as a neural network that learned to replicate examples without understanding the underlying problem might behave unpredictably when it encounters new settings.

Primary Area: Theory->Deep Learning

Keywords: Out-of-distribution generalization, Algorithm discovery, Deep learning theory, Mechanistic Interpretability

Submission Number: 16013