On diagonal approximations to the extended Kalman filter for online training of Bayesian neural networks

Peter G. Chang, Kevin Patrick Murphy, Matt Jones

Keywords: Bayesian inference, variational inference, online learning, extended Kalman filter, deep neural networks, non-stationary distributions, continual learning

TL;DR: New insights into recursive variational Bayes for online DNN learning

Abstract:

We present two approaches to approximate online Bayesian inference for the parameters of DNNs. Both are based on diagonal Gaussian approximations and linearize the network at each step to ensure efficient computation.
The first approach optimizes the exclusive KL, KL(p,q); this amounts to matching the marginal mean and {\em precision} of p and q.
The second approach optimizes the inclusive KL, KL(q,p), which amounts to matching the marginal mean and {\em variance} of p and q. The latter approach turns out to be equivalent to the previously proposed ``fully decoupled EKF'' approach.
We show experimentally that exclusive KL is more effective than both inclusive KL and one-pass SGD.