[RE] CNN-generated images are surprisingly easy to spot... for now

Joel Frank, Thorsten Holz

Keywords: Deepfake detection, CNN, Reproducibility, data augmentation

Abstract:

Reproducibility Summary

This work evaluates the reproducibility of the paper ``CNN-generated images are surprisingly easy to spot... for now'' by~\citeauthor{wang2019cnngenerated} published at CVPR 2020.
The paper addresses the challenge of detecting CNN-generated imagery, which has reached the potential to even fool humans.
The authors propose two methods which help an image classifier to generalize from being trained on one specific CNN to detecting imagery produced by unseen architectures, training methods, or data sets.

Scope of Reproducibility

The paper proposes two methods to help a classifier generalize: (i) using a diverse data set and (ii) utilizing different kinds of data augmentations.
This report focuses on assessing if these techniques indeed help the generalization process and if the results extend beyond the original contribution.

Methodology

We decided to implement the methods from scratch to highlight possible hurdles for practitioners who want to adapt the results.
For our experiments, we utilized the training and test data provided by the authors, as well as, creating our own data set to analyze how the proposed techniques perform on different data sets.
Note that overall we estimate the entire experiments to take around 380 GPU hours.

Results

In general, we were able to replicate the results reported in the paper but we also obtained results different from the paper.
After further investigations, we discovered that we had misunderstood one of the proposed data augmentation methods based on the description in the paper.
This misunderstanding lead to a different implementation and the discrepancies.
However, the overall trends of our experiments matched the original publication, i.e., data diversity and augmentations help generalization.
Thus, we believe both techniques to be effective.
We also performed several additional experiments on other data sets, highlighting limitations and clarifying the methods.

What was Easy

The paper is very detailed and we could faithfully replicate the experiments (bar the aforementioned exception).
We only utilized the authors' code sparingly, but the repository is very well documented and provides pre-trained models.

What was Difficult

The naming of one of the augmentations caused confusion on our part.
Note that the description in the paper is correct, however, the naming suggest the method works different than actually implemented.
This resulted in us implementing a variation of the proposed technique.
Surprisingly, our method actually improved on the original results.

Communication with Original Authors

Most of our question could be resolved by comparing our implementation against the authors' code.
We contacted the first author regarding our different results, he was very responsive and answered all of our questions.

Paper Url: https://openreview.net/forum?id=xXugzH_kllz&referrer=%5BML%20Reproducibility%20Challenge%202020%5D(%2Fgroup%3Fid%3DML_Reproducibility_Challenge%2F2020)