Reimplementing Fairness by Learning Orthogonal Disentangled Representations

Jeroen Pieter Jagt, Siem Teusink, Pim Wilbert Meerdink, Paul Lodder

Keywords: FAIR, variational inference, representation learning, disentanglement

Abstract:

Sarhan et al. [2020] propose a method of learning representations that can be used in downstream tasks, yet that are independent of certain sensitive attributes, such as race or sex. The learned representations can be considered “fair” as they are independent of sensitive attributes. The authors report results on five different datasets, which most notably include (1) the ability of the representations to be used for downstream tasks (target prediction accuracy) and (2) the extent to which sensitive information is present in these representations (sensitive prediction accuracy). In this text we report and compare the obtained results as well as highlight any difficulties encountered in reproducing the reference paper by Sarhan et al. [2020].

Methodology: As there was no openly available code base, we re-implemented the work. We included scripts to automatically download the required data, designed the dataloaders, and implemented the models as described in the reference paper. Code is available https://github.com/paulodder/fact2021.

Results: We were able to reproduce some of the results of the paper, but a significant part of our results was inconsistent with the findings of Sarhan et al. [2020]. For some of the simpler datasets we found similar patterns, but for the more complex tasks the models training became unstable, leading to results that varied significantly across random seeds. This made reproduction infeasible.

What was easy and what was difficult: Conceptually, the paper was interesting and, given some prior knowledge on Variational Autoencoders and the math involved, it was also relatively straightforward to understand. The most difficult aspect of the project was dealing with missing information. Many essential implementation details were missing, and there were inconsistencies in the pseudo-code provided. Resolving these issues provided significant difficulties.

Communication with original authors: Various emails were exchanged with the original authors, in which we received explanation about unclear aspects of the paper. In general, the authors were very helpful, but despite the fact that a few emails were exchanged, some aspects of the paper still remained unclear.

Paper Url: https://openreview.net/forum?id=uGujCu6SEH3