Bayesian ying-yang supervised learning, modular models, and three layer nets

Lei Xu

Abstract: Bayesian ying-yang (BYY) supervised learning system and theory is further re-elaborated, and the previous results of its uses on mixture-of-expert models, radial basis functions and three layer nets are systematically summarized. Moreover, new results on three layer net are presented. Using Taylor expansion on the distribution of the output layer, we find that maximum likelihood (ML) learning on a net with a probabilistic hidden layer is equivalent to adding a regularization to its counterpart with a deterministic hidden layer, which leads us not only an adaptive EM-like algorithm for ML learning on three layer net, but also a new type of regularization technique. Furthermore, an improved BYY criterion is obtained for selecting the number of hidden units.