Detective SAM: Adapting SAM to Localize Diffusion-based Forgeries via Embedding Artifacts

Gert Lek, Chaoyi Zhu, Pin-Yu Chen, Robert Birke, Lydia Y. Chen

Keywords: Segment Anything Model (SAM), Image Forgery Localization, Diffusion-Based Editing, Foundation Models, Learnable Prompts, Forensic Perturbation Signals

TL;DR: Detective SAM extends the Segment Anything Model with blur-driven forensic embedding signals, hierarchical learnable prompts, and lightweight adapters to accurately localize diffusion-based image forgeries.

Abstract:

Image forgery localization in the diffusion era
poses new challenges as modern editing pipelines
produce photorealistic, semantically coherent manipulations that bypass conventional detectors.
While some recent methods leverage foundation
model cues or handcrafted noise residuals, they
still miss the subtle embedding artifacts introduced by modern diffusion pipelines. In response,
we develop Detective SAM, which extends the
Segment Anything Model by incorporating a blur based detection signal, learnable coarse-to-fine
prompt generation, and lightweight fine-tuning
for automatic forgery mask generation. Detective SAM localizes forgeries with high precision.
On three challenging benchmarks (MagicBrush,
CoCoGlide, and IMD2020), it outperforms prior
state-of-the-art methods, demonstrating the power
of combining explicit forensic perturbation cues
with foundation-model adaptation for robust image forgery localization in the diffusion era. The code will be published in the anonymous repository https://anonymous.4open.science/r/DetectiveSAM-BC61 .

Submission Number: 52