Minimum Description Length Recurrent Neural Networks

Nur Lan, Emmanuel Chemla, Roni Katzir

Keywords: recurrent neural network, neural network, language modeling, minimum description length, genetic algorithm, semantics, syntax

Abstract: Recurrent neural networks (RNNs) face two well-known challenges: (a) the difficulty of such networks to generalize appropriately as opposed to memorizing, especially from very short input sequences (generalization); and (b) the difficulty for us to understand the knowledge that the network has attained (transparency). We explore the implications to these challenges of employing a general search through neural architectures using a genetic algorithm with Minimum Description Length (MDL) as an objective function. We find that MDL leads the networks to reach adequate levels of generalization from very small corpora, improving over backpropagation-based alternatives. We demonstrate this approach by evolving networks which perform tasks of increasing complexity with absolute correctness. The resulting networks are small, easily interpretable, and unlike classical RNNs, are provably appropriate for sequences of arbitrary length even when trained on very limited corpora. One case study is addition, for which our system grows a network with just four cells, reaching 100% accuracy (and at least .999 certainty) for arbitrary large numbers.

One-sentence Summary: Description length-based optimization of recurrent neural networks leads to interpretable and absolutely-correct networks for language modeling, using very small training data.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2111.00600/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=yYPWSeLkFQ