Replication Study of DECAF: Generating Fair Synthetic Data Using Causally-Aware Generative Networks

Shuai Wang, Paul Verhagen, Jennifer Zhuge, Velizar Shulev

Keywords: GAN, fairness, data utility, bias

TL;DR: Reproduction study of "DECAF: Generating Fair Synthetic Data Using Causally-Aware Generative Networks" by Breugel et al

Abstract:

Scope of reproducibility
In this paper, we attempt to reproduce the results found in "DECAF: Generating Fair Synthetic Data Using Causally-Aware Generative Networks" by Breugel et al. The goal of the original paper creates a model that intakes a biased dataset and outputs a debiased synthetic dataset that can be used to train downstream models to make unbiased predictions both on synthetic and real data.

Methodology
We built upon the incomplete code provided by the authors to repeat the first experiment which involves removing existing bias from real data with existing bias, and the second experiment where synthetically injected bias is added to real data and then remove by DECAF.

Results
We reproduced most of the data utility results reported in the first experiment for the Adult dataset. However, the fairness metric generally matches the original paper but is numerically not comparable in absolute or relative terms. For the second experiment, we were unsuccessful in reproducing the results found by the authors. We note however that we made considerable changes to the experimental setup, which may make it difficult to perform a direct comparison of the results.

What was easy
The smaller size and tabular format of both datasets allowed for quick training and model modifications.

What was difficult
There are several possible interpretations of the paper on both a methodological and conceptual level. Reproducing the experiments required rewriting or adding large sections of code. Given these multiple interpretations, it was difficult to be confident in the reproduction. In addition, several results found by the authors appear to be counterintuitive, such as algorithms debiasing without being designed to do so and sometimes outperforming debiasing algorithms on the same dataset.

Communication with original authors
We sent two emails to the authors describing our issues. We received a reply with a few extra files, but no direct answer to content questions.

Paper Url: https://openreview.net/forum?id=XN1M27T6uux

Paper Venue: NeurIPS 2021