\def \codeURL{https://github.com/YasBenAll/fact-ai-project}
\def \codeDOI{https://doi.org/10.5281/zenodo.7916507}
\def \dataURL{}
\def \dataDOI{}
\def \editorNAME{}
\def \editorORCID{}
\def \reviewerINAME{Anonymous Reviewers}
\def \reviewerIORCID{}
\def \reviewerIINAME{}
\def \reviewerIIORCID{}
\def \dateRECEIVED{03 February 2023}
\def \dateACCEPTED{}
\def \datePUBLISHED{}
\def \articleTITLE{[RE] Fairness Guarantees under Demographic Shift}
\def \articleTYPE{Replication}
\def \articleDOMAIN{ML Reproducibility Challenge 2022}
\def \articleBIBLIOGRAPHY{bibliography.bib}
\def \articleYEAR{2023}
\def \reviewURL{}
\def \articleABSTRACT{Scope of Reproducibility — The original authors’ main contribution is the family of Shifty algorithms, which can guarantee that certain fairness constraints will hold with high confidence even after a demographic shift in the deployment population occurs. They claim that Shifty provides these high‐confidence fairness guarantees without a loss in model performance, given enough training data. Methodology — The code provided by the original paper was used, and only some small adjustments needed to be made in order to reproduce the experiments. All model specifications and hyperparameters from the original implementation were used. Extending beyond reproducing the original paper, we investigated the sensibility of Shifty to the size of the bounding intervals limiting the possible demographic shift, and ran shifty with an additional optimization method. Results — Our results approached the results reported in the original paper. They supported the claim that Shifty reliably guarantees fairness under demographic shift, but could not verify that Shifty performs at no loss of accuracy. What was easy — The theoretical framework laid out in the original paper was well explained and supported by additional formulas and proofs in the appendix. Further, the authors provided clear instructions on how to run the experiments and provided necessary hyperparameters. What was difficult — While an open‐source implementation of Shifty was provided and was debugged with relatively low time investment, the code did not contain extensive documentation and was complex to understand. It was therefore difficult to verify that each part of the code functions as expected and to expand upon the existing experiments. Further, certain hyperparameter and model specifications deviated between the provided code and the original paper, which made it challenging to know which specifications to apply when reproducing. Communication with original authors — The first author of the original paper was contacted, but unfortunately we have yet to receive a reply.}
\def \replicationCITE{S. Giguere, B. Metevier, B. Castro da Silva, Y. Brun, P. Thomas, and S. Niekum. “Fairness guarantees under demographic shift.” In: International Conference on Learning Representations. 2022.}
\def \replicationBIB{giguere2022}
\def \replicationURL{https://openreview.net/pdf?id=wbPObLm6ueA}
\def \replicationDOI{}
\def \contactNAME{Valentin Leonhard Buchner}
\def \contactEMAIL{valentin.buchner@student.uva.nl}
\def \articleKEYWORDS{Fairness and Bias in ML, Fair Machine Learning, Reproduction, rescience c, machine learning}
\def \journalNAME{ReScience C}
\def \journalVOLUME{9}
\def \journalISSUE{2}
\def \articleNUMBER{1}
\def \articleDOI{}
\def \authorsFULL{Valentin Leonhard Buchner, Philip Onno Olivier Schutte, Yassin Ben Allal, and Hamed Ahadi}
\def \authorsABBRV{V.L. Buchner, P.O.O. Schutte, Y. Ben Allal, and H. Ahadi}
\def \authorsSHORT{Buchner et al.}
\title{\articleTITLE}
\date{03 February 2023}
\author[1,2,\orcid{0000-0002-1262-3016}]{Valentin Leonhard Buchner}
\author[1,\orcid{0009-0005-6198-2743}]{Philip Onno Olivier Schutte}
\author[1,2\orcid{0009-0006-9583-1705}]{Yassin Ben Allal}
\author[1,2\orcid{0000-0002-3277-3644}]{Hamed Ahadi}
\affil[1]{University of Amsterdam, The Netherlands}
\affil[2]{Vrije Universiteit Amsterdam, The Netherlands}
