MOLM: Mixture of LoRA Markers

Samar Fares, Nurbek Tastan, Noor Hazim Hussein, Karthik Nandakumar

Keywords: Watermarking, Diffusion models

Abstract: Generative models can generate photorealistic images at scale. This raises serious concerns about the ability to detect synthetically generated images and attribute these images to specific sources. While watermarking has emerged as a possible solution, existing methods remain fragile to realistic distortions, susceptible to adaptive removal, and expensive to update when the underlying watermarking key changes. We propose a general watermarking framework that formulates the encoding problem as key-dependent perturbation of the parameters of a generative model. Within this framework, we introduce Mixture of LoRA Markers (MOLM), a routing-based instantiation in which binary keys activate lightweight low-rank adapters (LoRA) inside residual and attention blocks. This design avoids key-specific re-training and achieves the desired properties such as imperceptibility, fidelity, verifiability, and robustness. Experiments on Stable Diffusion and FLUX show that MOLM preserves image quality while achieving robust key recovery against distortions, compression and regeneration, averaging attacks, and black-box adversarial attacks on the extractor. Code is available at [https://github.com/Samar-Fares/MOLM-Watermark](https://github.com/Samar-Fares/MOLM-Watermark)

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 9868