Identifying and Reducing "Generative Collisions" in Black-Box Large Language Models

Halley Young, Ashton Anderson, Siddhartha Sen

Keywords: diversity, LLMs, inference, black-box, creative

TL;DR: We formalize “generative collisions” and show that ORBIT—a black-box randomize-then-orthogonalize sampler with private buffers—wins 11/11 tasks in terms of avoiding near-duplicate responses to creative prompts

Abstract:

Large language models (LLMs) often return near‑duplicate responses to independent users who issue the same prompt even in tasks that demand creativity and uniqueness, a failure mode we call \textbf{generative collisions}. Existing "diverse decoding" techniques---temperature tuning, nucleus sampling, random prompt paraphrasing, etc.---spread a \textit{single} user’s samples but do not effectively prevent "collisions" \textit{across} users, sessions, or queries, because every independent sample still draws from the same high‑probability basin of the model’s distribution. This phenomenon is so pervasive that users frequently complain of an output "sounding like ChatGPT", implying a homogenization of writing style and culture.

To minimize generative collisions, we introduce \textbf{ORBIT} ("Orthogonal Randomized Buffer Inference Technique"), a black‑box algorithm that formalizes and combines two approaches for diverse generation: randomization and orthogonalization. ORBIT first initializes a small buffer of completions that are generated to optimize randomness over any other metric (including quality), and then samples the final output(s) to be as divergent from the existing buffer as possible while also maintaining quality. The buffer is instantiated locally and independently for each user session, requiring no cross-user coordination, and yet still minimizes collisions across sessions. ORBIT does not need access to model internals, and is therefore practical for commercial LLM endpoints.
We evaluate ORBIT on 11 tasks---ranging from regex‑scorable toy problems which don't require subjective evaluations to open‑ended creative writing that truly demands authenticity---and show that it consistently decreases collisions against all other black-box methods tested, lowering empirical collision rates by 1–2 orders of magnitude relative to all baselines. ORBIT's implementation is domain agnostic---it accepts a domain name and the original prompt as input---allowing it to be easily generalized to any domain.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 18748