Dale meets Langevin: A Multiplicative Denoising Diffusion Model

TMLR Paper9249 Authors

Abstract: Exponentiated gradient descent (EGD), a biologically motivated optimization algorithm that respects Dale's law, results in log-normally distributed synaptic weights, in alignment with experimental observations in neuroscience. Since the marginal distribution of geometric Brownian motion (GBM) at any fixed time is log-normal, there is a natural connection between EGD and GBM-based stochastic processes. We propose a multiplicative score-based generative model with GBM as a forward noising process and derive its corresponding reverse-time SDE in both the ambient space and in the $\log$-transformed space. We derive two multiplicative samplers by discretizing the corresponding reverse-time SDEs: a sign-agnostic sampler obtained directly from the ambient-space reverse-time SDE, and a sign-preserving sampler, which we refer to as the Dale-Langevin sampler, obtained via the Lamperti transform. We further connect the framework to Mirrored Langevin Dynamics, showing that the convex function driving EGD in optimization precisely governs the Dale-Langevin sampler. The Stein score, defined as $\nabla \log p_{\boldsymbol{X}}(\boldsymbol{x})$ for a random vector $\boldsymbol{X}$ with density $p_{\boldsymbol{X}}$ evaluated at $\boldsymbol{x}$, comes up naturally in the additive noise based diffusion models. In the multiplicative setting, we encounter $\boldsymbol{x} \circ \nabla \log p_{\boldsymbol{X}}(\boldsymbol{x})$, a modulated version of the Stein score for sampling, which we name the Hyvärinen score. In order to estimate the Hyvärinen score, we introduce the multiplicative denoising score-matching loss (M-DSM), the multiplicative explicit score-matching loss (M-ESM), and establish their equivalence. This development subsumes the non-negative score matching loss of Hyvärinen (2007) as a special case. Experimental results on MNIST, Fashion-MNIST, Kuzushiji MNIST, and CIFAR-10 validate the generative capability of the proposed framework.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Valentin_De_Bortoli1

Submission Number: 9249