Improving the Neural GPU Architecture for Algorithm Learning

Anonymous

Abstract:

Algorithm learning is a core problem in artificial
intelligence with significant implications on automation
level that can be achieved by machines.
Recently deep learning methods are emerging for
synthesizing an algorithm from its input-output
examples, the most successful being the Neural
GPU, capable of learning multiplication. We
present several improvements to the Neural GPU
that substantially reduces training time and improves
generalization. We introduce a new technique
- hard nonlinearities with saturation costs -
that has general applicability. We also introduce
a technique of diagonal gates that can be applied
to active-memory models. The proposed architecture
is the first capable of learning decimal
multiplication end-to-end.

Keywords: Algorithm learning, Neural GPU, convolution

TL;DR: We present several improvements to the Neural GPU that substantially reduces training time and improves generalization

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:1702.08727/code)