Training restricted Boltzmann Machine with dynamic learning rate

Linkai Luo, Yudan Wang, Hong Peng, Zhimin Tang, Shiyang You, Xiaoqin Huang

Abstract: Restricted Boltzmann Machine (RBM) has been successfully applied to many different machine learning and pattern recognition problems. Usually, fixed learning rate (FLR) is used for training RBM. However, the reconstruction error (RCERR) with FLR may not be declined each iteration, which will result in a slow convergence speed. In this paper, we propose a method to dynamically choose the learning rate by reducing RCERR properly. The experiments on MNIST database and Caltech 101 Silhouettes database show the RBMs trained with dynamic learning rate (DLR) are better than that trained with FLR in classification accuracy and stability. It indicates DLR may be more suitable for training RBM.