Effects of Noise on Convergence and Generalization in Recurrent Networks

Kam-Chuen Jim, Bill G. Horne, C. Lee Giles

Abstract: We introduce and study methods of inserting synaptic noise into dynamically-driven recurrent neural networks and show that ap(cid:173) plying a controlled amount of noise during training may improve convergence and generalization. In addition, we analyze the effects of each noise parameter (additive vs. multiplicative, cumulative vs. non-cumulative, per time step vs. per string) and predict that best overall performance can be achieved by injecting additive noise at each time step. Extensive simulations on learning the dual parity grammar from temporal strings substantiate these predictions.