A reproducibility study of “User-item fairness tradeoffs in recommendations”

Sander Honig, Elyanne Oey, Lisanne Wallaard, Sharanda Suttorp, Clara Rus

Abstract: Recommendation systems are necessary to filter the abundance of information presented in our everyday lives. A recommendation system could exclusively recommend items that users prefer the most, potentially resulting in certain items never getting recommended. Conversely, an exclusive focus on including all items could hurt overall recommendation quality. This gives rise to the challenge of balancing user and item fairness. The paper “User-item fairness tradeoffs in recommendations” by Greenwood et al. (2024) explores this tradeoff by developing a theoretical framework that optimizes for user-item fairness constraints. Their theoretical framework suggests that the cost of item fairness is low when users have varying preferences compared to each other, and may be high for users whose preferences are misestimated. They empirically measured these phenomena by creating their own recommendation system on arXiv preprints, and confirmed that the cost of item fairness is low when users have preferences that differ from one another. However, contrary to their theoretical expectations, misestimated users do not encounter a higher cost of item fairness. This study investigates the reproducibility of their research by replicating the empirical study. Additionally, we extend their research in two ways: (i) verifying the generalizability of their findings on a different dataset (Amazon books reviews), and (ii) analyzing the tradeoffs when recommending multiple items to a user instead of a single item. Our results further validate the claims made in the original paper. We concluded the claims hold true when recommending multiple items, with the cost of item fairness decreasing as more items are recommended.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

We would like to thank the reviewers for their valuable feedback. Upon the action editor's request, we made the following changes to section 3.3.3:
- Clarified in the caption of each figure that U^{*} is plotted.
- Rewritten section 3.3.3 to clarify our approach and what was meant with sampling.

Video: https://www.youtube.com/watch?v=I5NTCq3oqDs

Code: https://github.com/sanderhonig/RE-User-item-fairness-tradeoffs-in-recommendations

Assigned Action Editor: ~Dennis_Wei1

Submission Number: 4303