Computing Low-Entropy Couplings for Large-Support Distributions

Samuel Sokota, Dylan Sam, Christian Schroeder de Witt, Spencer Compton, Jakob Nicolaus Foerster, J Zico Kolter

Keywords: minimum-entropy coupling, steganography

TL;DR: Method for computing low-entropy couplings for arbitrary large-support discrete distributions

Abstract: Minimum-entropy coupling (MEC)---the process of finding a joint distribution with minimum entropy for given marginals---has applications in areas such as causality and steganography. However, existing algorithms are either computationally intractable for large-support distributions or limited to specific distribution types and sensitive to hyperparameter choices. This work addresses these limitations by unifying a prior family of iterative MEC (IMEC) approaches into a generalized partition-based formalism. From this framework, we derive a novel IMEC algorithm called ARIMEC, capable of handling arbitrary discrete distributions, and introduce a method to make IMEC robust to suboptimal hyperparameter settings. These innovations facilitate the application of IMEC to high-throughput steganography with language models, among other settings.

List Of Authors: Sokota, Samuel and Sam, Dylan and Schroeder de Witt, Christian and Compton, Spencer and Foerster, Jakob and Kolter, J Zico

Code Url: https://github.com/ssokota/mec

Submission Number: 474