Exploring the Learning Mechanisms of Neural Division Modules

Bhumika Mistry, Katayoun Farrahi, Jonathon Hare

Abstract:

Of the four fundamental arithmetic operations (+, -, $\times$, $\div$), division is considered the most difficult for both humans and computers.
In this paper, we show that robustly learning division in a systematic manner remains a challenge even at the simplest level of dividing two numbers. We propose two novel approaches for division which we call the Neural Reciprocal Unit (NRU) and the Neural Multiplicative Reciprocal Unit (NMRU), and present improvements for an existing division module, the Real Neural Power Unit (Real NPU). In total we measure robustness over 475 different training sets for setups with and without input redundancy. We discover robustness is greatly affected by the input sign for the Real NPU and NRU, input magnitude for the NMRU and input distribution for every module. Despite this issue, we show that the modules can learn as part of larger end-to-end networks.

Submission Length: Long submission (more than 12 pages of main content)

Changes Since Last Submission: Camera-ready submission; deanonymized paper.

Code: https://github.com/bmistry4/nalm-division

Assigned Action Editor: ~Caglar_Gulcehre1

License: Creative Commons Attribution 4.0 International (CC BY 4.0)

Submission Number: 159