Sample Space Truncation on Boltzmann Machines

Mahito Sugiyama, Koji Tsuda, Hiroyuki Nakahara

Keywords: Boltzmann machines, log-linear model, bias-variance decomposition, dually flat manifold

TL;DR: Efficient training of Boltzmann machines via sample space truncation and its theoretical analysis using information geometry.

Abstract: We present a lightweight variant of Boltzmann machines via sample space truncation, called a truncated Boltzmann machine (TBM), which has not been investigated before while can be naturally introduced from the log-linear model viewpoint. TBMs can alleviate the massive computational cost of exact training of Boltzmann machines that requires exponential time evaluation of expected values and the partition function of the model distribution. To analyze the learnability of TBMs, we theoretically provide bias-variance decomposition of the log-linear model using dually flat structure of statistical manifolds.