Better Training using Weight-Constrained Stochastic Dynamics

Benedict J. Leimkuhler, Tiffany J. Vlaar, Timothée Pouchon, Amos J. Storkey

Abstract: We employ constraints to control the parameter space of deep neural networks throughout training. The use of customised, appropriately designed constraints can reduce the vanishing/exploding gradie...