Replication study of "Data-Driven Methods for Balancing Fairness and Efficiency in Ride-Pooling"

Vera Neplenbroek, Sabijn Perdijk, Victor Prins

Keywords: ridesharing, fairness, reproduction

Abstract:

Scope of Reproducibility

We evaluate the following claims related to fairness-based objective functions presented in "Data-Driven Methods for Balancing Fairness and Efficiency in Ride-Pooling" (Raman et al., 2021): (1) For the four objective functions, the success rate in the worst-off neighborhood increases monotonically with respect to the overall success rate. (2) The proposed objective functions do not lead to a higher income for the lowest-earning drivers, nor a higher total income, compared to a request-maximizing objective function. (3) The driver-side fairness objective can outperform a request-maximizing objective in terms of overall success rate and success rate in the worst-off neighborhood. This means that this objective, whilst reducing the spread of income, also positively impacts rider fairness and profitability.

Methodology

The code provided by the authors was used as a base for our re-implementation in PyTorch. We evaluate the claims by the original authors by (a) replicating their experiments, (b) testing for sensitivity to a different value estimator, (c) examining sensitivity to changes in the preprocessing method, and (d) testing for generalizability by applying their method to a different dataset.

Results

We reproduced the first claim since we observed the same monotonic increase of the success rate in the worst-off neighborhood with respect to the overall success rate. The second claim we did not reproduce, since we found that the driver-side fairness objective function obtains a higher income for the lowest-earning drivers than the request-maximizing objective function. We reproduced the third claim, since the driver-side objective function performs best in terms of overall success rate and success rate in the worst-off neighborhood, and also reduces the spread of income. Changes of the value estimator, preprocessing method and even dataset all led to consistent results regarding these claims.

What was easy

The paper is written engagingly and the theoretical sections, in particular, give a clear description of the problem setup and objectives. The paper is also accompanied by an open-source code base, which supports reproduction efforts.

What was difficult

The provided code lacks a script to preprocess raw data, which is required to reproduce the experiment, nor was the preprocessed data openly available. Additionally, complex code structure and scarce commenting complicated replication.

Communication with original authors

Due to the absence of preprocessed data, we contacted the authors, who quickly provided the requested data.

Paper Url: https://www.ijcai.org/proceedings/2021/51

Paper Venue: IJCAI 2021