Detect any deepfakes: Segment anything meets face forgery detection and localization

Yingxin Lai, Zhiming Luo, Zitong YU

Abstract:

The rapid advancements in computer vision have stimulated
remarkable progress in face forgery techniques, capturing the dedicated
attention of researchers committed to detecting forgeries and precisely localizing manipulated areas. Nonetheless, with limited fine-grained pixelwise supervision labels, deepfake detection models perform unsatisfactorily on precise forgery detection and localization. To address this challenge, we introduce the well-trained vision segmentation foundation model,
i.e., Segment Anything Model (SAM) in face forgery detection and localization. Based on SAM, we propose the Detect Any Deepfakes (DADF)
framework with the Multiscale Adapter, which can capture short- and
long-range forgery contexts for efficient fine-tuning. Moreover, to better identify forged traces and augment the model’s sensitivity towards
forgery regions, Reconstruction Guided Attention (RGA) module is proposed. The proposed framework seamlessly integrates end-to-end forgery
localization and detection optimization. Extensive experiments on three
benchmark datasets demonstrate the superiority of our approach for both
forgery detection and localization. The codes will be released soon at
https://github.com/laiyingxin2/DADF
.