# AdvPlot

## Introduction

This repository contains the code required to produce the plots from the paper
`Unbiased Decisions Reduce Regret: Adversarial Optimism for the Bank Loan Problem'.

## How To Run

Experiments can be run using `experiments.py` or multiple experiments can be 
submitted in parallel to a cluster using slurm by running `slurm.py`. There are a 
number of customisable hyperparameters at the start of each of these files, 
which dictate the experiments to be run:
1. `METHODS`: A list indicating which algorithms to run. The possible choices for this
list are dictated by the function `algo_to_params` in `experiments.py`. Currently, the
possible choices are those algorithms included in the paper, `Eps_Greedy`, `Greedy`, 
`NeuralUCB`, `PLOT`, AdversarialPLOT with 1 epoch of adversarial training and not resetting the 
network at each step of PLOT (denoted `AdvPLOT_1e_nor`), and plain Adversarial with 1 epoch and 
not resetting (denoted `Adv_1e_nor`). Further algorithms can be specified by
customising `exploration_hparams` in `algo_to_params`. 
2. `DATASETS`: A list of which datasets to compare the algorithms in `METHODS` against. 
The currently implemented choices are Adult, Bank, Crime, German, and MNIST.
3. `TIMESTEPS`: The number of timesteps to run each algorithm on each dataset. 
4. `SEED`: The initial random seed.

The following two hyperparameters are specific to slurm, and for running multiple repeats of
experiments in parallel:
5. `NUM_EXPERIMENTS`: The number of times to repeat each experiment.
5. `NUM_SEEDS`: The number of random seeds to run each experiment for.

Independent to these hyperparameters is the choice of protected characteristic to monitor
during the runs. This can be modified in `algo_to_params` on line 38 of `experiments.py`.

## How To Plot

If running experiments via `experiments.py`, then plots of the outputs will be saved.
If the experiments are run via `slurm.py`, then `aggregate_and_plot.py` is used to plot the data.
It is important to set the hyperparameters at the top of `aggregate_and_plot.py` to 
be the same as those used when running `slurm.py`. Running 
`aggregate_and_plot.py` will generate plots of regret and protected characteristics
averaged across all the seeds experiments were run for.
