Steepest descent as message passing

Justin Dauwels, Sascha Korl, Hans-Andrea Loeliger

Abstract: It is shown how steepest descent (or steepest ascent) may be viewed as a message passing algorithm with "local" message update rules. For example, the well-known backpropagation algorithm for the training of feedforward neural networks may be viewed as message passing on a factor graph. The factor graph approach with its emphasis on "local" computations makes it easy to combine steepest descent with other message passing algorithms such as the sum/max-product algorithms, expectation maximization, Kalman filtering/smoothing, and particle filters. As an example, parameter estimation in a state space model is considered. For this example, it is shown how steepest descent can be used for the maximization step in expectation maximization.