On The Representation Properties Of The Perturb-Softmax And The Perturb-Argmax Probability Distributions
Keywords: representation properties, Gumbel-Softmax, Gumbel-Argmax, minimality, completeness, discrete probabilistic models
Abstract: The Gumbel-Softmax probability distribution allows learning discrete tokens in generative learning, whereas the Gumbel-Argmax probability distribution is useful in learning discrete structures in discriminative learning. Despite the efforts invested in optimizing these models, their properties are underexplored. In this work, we investigate their representation properties and determine for which families of parameters these probability distributions are complete, that is, can represent any probability distribution, and minimal, i.e., can represent a probability distribution uniquely. We rely on convexity and differentiability to determine these conditions and extend this framework to general probability models, denoted Perturb-Softmax and Perturb-Argmax. We conclude the analysis by identifying two sets of parameters that satisfy these assumptions and thus admit a complete and minimal representation. A faster convergence rate of Gaussian-Softmax in comparison to Gumbel-Softmax further motivates our study, as the experimental evaluation validates.
Supplementary Material: zip
Primary Area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 4453
Loading