On diagonal approximations to the extended Kalman filter for online training of Bayesian neural networks![Download PDF](/images/pdf_icon_blue.svg)
Keywords: Bayesian inference, variational inference, online learning, extended Kalman filter, deep neural networks, non-stationary distributions, continual learning
TL;DR: New insights into recursive variational Bayes for online DNN learning
Abstract: We present two approaches to approximate online Bayesian inference for the parameters of DNNs. Both are based on diagonal Gaussian approximations and linearize the network at each step to ensure efficient computation.
The first approach optimizes the exclusive KL, KL(p,q); this amounts to matching the marginal mean and {\em precision} of p and q.
The second approach optimizes the inclusive KL, KL(q,p), which amounts to matching the marginal mean and {\em variance} of p and q. The latter approach turns out to be equivalent to the previously proposed ``fully decoupled EKF'' approach.
We show experimentally that exclusive KL is more effective than both inclusive KL and one-pass SGD.
1 Reply
Loading