Detecting Deep-Fake Videos from Appearance and Behavior

Shruti Agarwal, Hany Farid, Tarek El-Gaaly, Ser-Nam Lim

Abstract: Synthetically-generated audios and videos - so-called deep fakes - continue to capture the imagination of the computer-graphics and computer-vision communities. At the same time, the democratization of access to technology that can create a sophisticated manipulated video of anybody saying anything continues to be of concern because of its power to disrupt democratic elections, commit small to large-scale fraud, fuel disinformation campaigns, and create non-consensual pornography. We describe a biometric-based forensic technique for detecting face-swap deep fakes. This technique combines a static biometric based on facial recognition with a temporal, behavioral biometric based on facial expressions and head movements, where the behavioral embedding is learned using a CNN with a metric-learning objective function. We show the efficacy of this approach across several large-scale video datasets, as well as in-the-wild deep fakes.