Convergence of the Wake-Sleep Algorithm

Shiro Ikeda, Shun-ichi Amari, Hiroyuki Nakahara

Abstract: The W-S (Wake-Sleep) algorithm is a simple learning rule for the models with hidden variables. It is shown that this algorithm can be applied to a factor analysis model which is a linear version of the Helmholtz ma(cid:173) chine. But even for a factor analysis model, the general convergence is not proved theoretically. In this article, we describe the geometrical un(cid:173) derstanding of the W-S algorithm in contrast with the EM (Expectation(cid:173) Maximization) algorithm and the em algorithm. As the result, we prove the convergence of the W-S algorithm for the factor analysis model. We also show the condition for the convergence in general models.