The Gaussian-Multinoulli Restricted Boltzmann Machine: A Potts Model Extension of the GRBM

Nikhil Kapasi, Mohamed Elfouly, William Whitehead, Luke Theogarajan

Keywords: Probabilistic Model, RBM, Generative Models

TL;DR: GM-RBM replaces GB-RBM’s binary hiddens with q-state Potts slots and, in capacity/parameter-matched tests, delivers stronger associative recall and competitive generation with Gibbs, showing when categorical latents beat binary.

Abstract: Many real-world tasks, from associative memory to symbolic reasoning, benefit from discrete, structured representations that standard continuous latent models can struggle to express. We introduce the Gaussian–Multinoulli Restricted Boltzmann Machine (GM-RBM), a generative energy-based model that extends the Gaussian–Bernoulli RBM (GB-RBM) by replacing binary hidden units with $q$-state categorical (Potts) units, yielding a richer latent state space for multivalued concepts. We provide a self-contained derivation of the energy, conditional distributions, and learning rules, and detail practical training choices (contrastive divergence with temperature annealing and intra-slot diversity constraints) that avoid state collapse. To separate architectural effects from sheer latent capacity, we evaluate under both capacity-matched and parameter-matched setups, comparing GM-RBM with GB-RBM configured to have the same number of possible latent assignments. On analogical recall and structured memory benchmarks, GM-RBM achieves competitive, and in several regimes, improved recall at equal capacity with comparable training cost, despite using only Gibbs updates. The discrete $q$-ary formulation is also amenable to efficient implementation. These results clarify when categorical hidden units provide a simple, scalable alternative to binary latents for discrete inference within tractable RBMs.

Primary Area: generative models

Submission Number: 16508