[Re] Reproducibility Study of Equal Improvability Fairness Notion
Abstract: Our research validates and expands the Equal Improvability (EI) framework, which aims to equalize acceptance rates across different groups by quantifying required improvement efforts, thereby enhancing long-term fairness. By replicating the original findings, we reaffirm the foundational claims of EI. Additionally, extended experiments are conducted to probe the efficacy of EI under varied scenarios. To enhance long-term fairness, we propose non-parametric updates and Chi-square fit to generalize the dataset, in contrast to the Gaussian distribution dataset from the original study. Our analysis shows that the EI framework struggles with adapting to the Chi-square fit and exhibits even poorer performance with non-parametric updates in long-term scenarios, indicating challenges in dynamic distribution scenarios. The update rule is modified to align more with theorem and intuition. It is proved that EI is more robust to noise compared with the other notions. The examination of varying decision fractions uncovers the conditional robustness of EI across different acceptance rates. These experiments highlight the strengths of EI in certain contexts and its limitations in others, providing a nuanced understanding of its applicability and areas for improvement in the pursuit of fairness in machine learning.
Submission Length: Regular submission (no more than 12 pages of main content)
Changes Since Last Submission: 1. Provided more background on EI, including definitions of symbols, equations and update rules.
2. Discussed the concept of long-term fairness and its evaluation methods compared to earlier experiments focusing on error rates and EI gap for a single frozen model.
3. Clarified the subsection on "Decision fraction" to improve comprehensibility and highlighted the main findings from that section.
4. Corrected the equation for EI disparity and revised the equation for long-term unfairness to account for the classifier's impact on the results.
5. Rectified the definitions of $x$ and $z$ to accurately reflect their roles in the paper.
6. Amended the definition of sensitive attribute to align with original paper, as well as immutable, manipulable and improvable features. To avoid confusion, the “sensitive feature” is changed to “sensitive attribute”, since they have the same meaning in our paper.
7. Added detailed descriptions of parameter updates and experimental rounds to provide background for subsequent discussions.
8. Checked and corrected symbol errors on page 7.
9. Refine the language in statements regarding EI fairness performance to ensure logical consistency.
10. Provided more detailed descriptions of the experimental setup and hyperparameters for long-term feature distribution and new experiments to explain differences from the original paper.
11. Revised the definition of EI disparity to align with the original paper.
12. Explained undefined symbols in equations and fixed formatting issues.
13. In the Table 4, "different value from original work" is further explained in Section 4.1, claim 1 to illustrate that the results are reproduced and almost the same.
14. Summarized the most important takeaways from replicating and expanding upon the original EI paper.
Assigned Action Editor: ~Andrew_Miller1
Submission Number: 2260
Loading