Strategic classification made practical: reproduction

Guilly Kolkman, Maks kulicki, Jan Athmer, Alex Labro

TL;DR: A reproduction of the original paper "strategic classification made practical"

Abstract:

Scope of Reproducibility
In this work, the paper Strategic Classification Made Practical is evaluated through a reproduction study. The results from the reproduction examines if the claims made in the paper are valid. We could find two main claims that were made by the authors that we will attempt to reproduce. Those are as follows:
1."We propose a novel learning framework for strategic classification that is practical, effective, and flexible.This allows for differentiation through strategic user responses, which supports end-to-end training."
2."We propose several forms of regularization that encourage learned models to promote favorable social outcomes."
We interpret practical, effective and flexible as such that the model should work better on a variety of real life problems than their non-strategic counterpart.

Methodology
In this paper, the same code, datasets and hyperparameters were used as the original paper to reproduce the results. To further validate the claims from the original paper, we extended the original implementation to include an experiment that tests performance on a dataset containing both strategic (also referred to as gaming) and non-strategic users.

Results
The reproduction of the original paper as well as the extended implementation were successful. We were able to reproduce the original results and examine the performance of the proposed model in an environment where strategic and non-strategic users both present. Linear models seem to struggle with different proportions of strategic users, while the non-linear model (RNN) achieves good performance regardless of the proportion of strategic users.

What was easy
The codebase for the paper was available on GitHub which meant that we didn’t have to start from scratch. They also provided us with the original data. The codebase also came with the original results from the authors which meant that comparing the results was easy.

What was difficult
Although the code was available, documentation of the code was quite sparse. Therefore, it was hard to figure out what each part of the code did and made it difficult to interpret what the results actually meant at certain stages.

Communication with original authors
The University of Amsterdam communicated before the course with the authors about the datasets. While working on the reproduction we sent one email about clarification of their method and to request a missing dataset.

Paper Url: https://arxiv.org/pdf/2103.01826.pdf

Paper Venue: ICML 2021

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/strategic-classification-made-practical/code)