Early Brain Damage

Volker Tresp, Ralph Neuneier, Hans-Georg Zimmermann

Abstract: Optimal Brain Damage (OBD) is a method for reducing the num(cid:173) ber of weights in a neural network. OBD estimates the increase in cost function if weights are pruned and is a valid approximation if the learning algorithm has converged into a local minimum. On the other hand it is often desirable to terminate the learning pro(cid:173) cess before a local minimum is reached (early stopping). In this paper we show that OBD estimates the increase in cost function incorrectly if the network is not in a local minimum. We also show how OBD can be extended such that it can be used in connec(cid:173) tion with early stopping. We call this new approach Early Brain Damage, EBD. EBD also allows to revive already pruned weights. We demonstrate the improvements achieved by EBD using three publicly available data sets.