Keywords: Latent space smoothing, LASSI, reproducibility, fairness, AI, deep learning
TL;DR: Reproducibility studies on a newly proposed and more fair representation learning method called LASSI
Abstract: - Scope of Reproducibility
The aim of this work is to study the reproducibility of the paper 'Latent Space Smoothing for Individually Fair Representations' by Peychev et al., in which a novel representation learning method called LASSI is proposed. We aim to verify the three main claims made in the original paper: (1) LASSI increases certified individual fairness, while keeping prediction accuracies high, (2) LASSI can handle various sensitive attributes and attribute vectors and (3) LASSI representations can achieve high certified individual fairness even when downstream tasks are not known. In addition, we aim to test the robustness of their claims by conducting additional experiments.
- Methodology
To reproduce the experiments, we use the step-by-step guidelines supplied by the original authors on their github repository. The experimental setup and datasets used are identical to the work reported in the original paper. We write additional code to run experiments beyond the scope of the work done by Peychev et al. In order to comply with resource limitations, we reproduce only the experiments relevant to the main claims. In total a budget of 45 hours on an NVIDIA Titan RTX GPU is used.
- Results:
We are able to reproduce and verify the three main claims of the original paper, by reproducing the results within 5% of the reported values. The additional experiments were succesful and strengthen the claims that LASSI increases certified individual fairness compared to the baseline models. Outliers of the experiments are studied and found to be caused by biased and inaccurate input data.
- What was easy:
Reproducing the original experiments was made possible by the extensive documentation and guidelines created by the authors in their code and public GitHub repository. The theoretical background provided in their paper was clear and detailed, allowing a deeper understanding about the inner workings of their models and metrics.
- What was difficult:
The main difficulty was found within the complex structure of the original code files and the related functions across these files. The code needed to perform our additional experiments was therefore also complex and required us to alter many different functions in the original code.
- Communication with original authors:
To keep the reproducibility report a fair assessment, this work has been sent to the original authors to ask for their feedback and comments.
Paper Url: https://arxiv.org/abs/2111.13650v3
Paper Venue: ECCV 2022
Confirmation: The report pdf is generated from the provided camera ready Google Colab script, The report metadata is verified from the camera ready Google Colab script, The report contains correct author information., The report contains link to code and SWH metadata., The report follows the ReScience latex style guides as in the Reproducibility Report Template (https://paperswithcode.com/rc2022/registration)., The report contains the Reproducibility Summary in the first page., The latex .zip file is verified from the camera ready Google Colab script
Latex: zip
Journal: ReScience Volume 9 Issue 2 Article 30
Doi: https://www.doi.org/10.5281/zenodo.8173725
Code: https://archive.softwareheritage.org/swh:1:dir:ebe7321bfcc8268ca48b1b269c64b6fe1df79653
0 Replies
Loading