Reproducibility Study of ’Latent Space Smoothing for Individually Fair Representations’

Didier Merk, Tsatsral Mendsuren, Denny Smit, Boaz Beukers

Keywords: Latent space smoothing, LASSI, reproducibility, fairness, AI, deep learning

TL;DR: Reproducibility studies on a newly proposed and more fair representation learning method called LASSI

Abstract:

- Scope of Reproducibility
The aim of this work is to study the reproducibility of the paper 'Latent Space Smoothing for Individually Fair Representations' by Peychev et al., in which a novel representation learning method called LASSI is proposed. We aim to verify the three main claims made in the original paper: (1) LASSI increases certified individual fairness, while keeping prediction accuracies high, (2) LASSI can handle various sensitive attributes and attribute vectors and (3) LASSI representations can achieve high certified individual fairness even when downstream tasks are not known. In addition, we aim to test the robustness of their claims by conducting additional experiments.

- Methodology
To reproduce the experiments, we use the step-by-step guidelines supplied by the original authors on their github repository. The experimental setup and datasets used are identical to the work reported in the original paper. We write additional code to run experiments beyond the scope of the work done by Peychev et al. In order to comply with resource limitations, we reproduce only the experiments relevant to the main claims. In total a budget of 45 hours on an NVIDIA Titan RTX GPU is used.

- Results:
We are able to reproduce and verify the three main claims of the original paper, by reproducing the results within 5% of the reported values. The additional experiments were succesful and strengthen the claims that LASSI increases certified individual fairness compared to the baseline models. Outliers of the experiments are studied and found to be caused by biased and inaccurate input data.

- What was easy:
Reproducing the original experiments was made possible by the extensive documentation and guidelines created by the authors in their code and public GitHub repository. The theoretical background provided in their paper was clear and detailed, allowing a deeper understanding about the inner workings of their models and metrics.

- What was difficult:
The main difficulty was found within the complex structure of the original code files and the related functions across these files. The code needed to perform our additional experiments was therefore also complex and required us to alter many different functions in the original code.

- Communication with original authors:
To keep the reproducibility report a fair assessment, this work has been sent to the original authors to ask for their feedback and comments.

Paper Url: https://arxiv.org/abs/2111.13650v3

Paper Venue: ECCV 2022

Confirmation: The report pdf is generated from the provided camera ready Google Colab script, The report metadata is verified from the camera ready Google Colab script, The report contains correct author information., The report contains link to code and SWH metadata., The report follows the ReScience latex style guides as in the Reproducibility Report Template (https://paperswithcode.com/rc2022/registration)., The report contains the Reproducibility Summary in the first page., The latex .zip file is verified from the camera ready Google Colab script

Journal: ReScience Volume 9 Issue 2 Article 30

Doi: https://www.doi.org/10.5281/zenodo.8173725

Code: https://archive.softwareheritage.org/swh:1:dir:ebe7321bfcc8268ca48b1b269c64b6fe1df79653