SELFI: Selective Fusion of Identity for Generalizable Deepfake Detection

Younghun Kim, Minsuk Jang, Myung-Joon Kwon, Wonjun Lee, Changick Kim

Keywords: Deepfake Detection, Identity Fusion, Generalization, Forgery Analysis, Cross-dataset Evaluation, Face Recognition, Robust AI

TL;DR: We propose SELFI, a deepfake detection framework that adaptively fuses face identity features based on manipulation context, achieving strong generalization by controlling identity bias.

Abstract: Face identity provides a remarkably powerful signal for deepfake detection. Prior studies have shown that even when not explicitly modeled, deepfake classifiers tend to implicitly learn identity features during training. This has led to two conflicting viewpoints in the literature: some works attempt to completely suppress identity cues to mitigate bias, while others rely on them exclusively as a strong forensic signal. To reconcile these opposing stances, we conduct a detailed empirical analysis based on two central hypotheses: (1) whether face identity alone is inherently discriminative for detecting deepfakes, and (2) whether such identity features generalize poorly across manipulation methods. Through extensive experimentation, we confirm that face identity is indeed a highly informative signal—but its utility is context-dependent. While some manipulation methods preserve identity-consistent artifacts, others distort identity cues in ways that can harm generalization. These findings suggest that identity features should not be suppressed or relied upon blindly. Instead, they should be explicitly modeled and adaptively controlled based on their per-sample relevance. To this end, we propose SELFI (SELective Fusion of Identity), a generalizable deepfake detection framework that dynamically modulates identity usage. SELFI consists of: (1) a Forgery-Aware Identity Adapter (FAIA) that explicitly extracts face identity embeddings from a frozen face recognition model and projects them into a forgery-relevant space using auxiliary supervision, and (2) an Identity-Aware Fusion Module (IAFM) that selectively integrates identity and visual features via a relevance-guided fusion mechanism. Extensive experiments on four benchmark datasets demonstrate that SELFI achieves strong generalization across manipulation methods and datasets, outperforming prior state-of-the-art methods by an average of 3.1% frame-level AUC in cross-dataset evaluations. Notably, on the challenging DFDC benchmark, SELFI improves over the previous best by a significant 6% margin, highlighting the effectiveness of adaptive identity control. The code will be released upon acceptance of the paper.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 10047