Categorical Drifting Models

Floor Eijkelboom, Nabil Iqbal, Max Welling, Jan-Willem van de Meent

Keywords: Generative modeling, discrete generation, drifting models, bregman divergence, dual

TL;DR: CatDrift turns drifting model into simplex-aware logit tilting, enabling teacher-free one-step generation for categorical images, molecules, and DNA.

Abstract: We propose CatDrift, a categorical extension of Drifting Models for one-step generation in discrete spaces. We reinterpret drifting as a local alignment problem: each generated sample is corrected in the direction of a data-dependent field while remaining close to its current value. In the original formulation, this constraint is Euclidean, yielding additive updates in sample space. This choice is natural for continuous data but ill-suited to categorical variables, whose states lie on the simplex. Replacing the Euclidean penalty with a Bregman divergence turns the drifting update into a mirror-descent step, a standard object in optimization with a natural information-geometric interpretation. For negative entropy, the categorical update has a closed form: build a kernel-weighted positive-minus-negative field from real samples and the generator’s own outputs, add it to logits, project with softmax, and regress to the frozen target by cross-entropy. The result is a simple categorical drifting algorithm that obtains competitive-to-state-of-the-art one-step results on discrete image generation, molecular generation, and DNA sequence generation, without distilling diffusion models.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 234