# The Disparate Benefits of Deep Ensembles

To change where the datasets are loaded, results are stored, etc. change paths in ```source/constants.py```

## Setting up Medical Imaging Experiments

First, follow the instructions given in [setup_chexpert.py](./setup_chexpert.py) for downloading raw data and run the file to create a downsized version of it for faster experiments and move the necessary csv files containing label and protected attribute information.
An account at https://stanfordaimi.azurewebsites.net is needed to download this dataset.

## Experimental Pipeline

First, all considered options for training ensembles (tasks, runs, models) have to be executed by running [train_face_detection_ensemble.py](./train_face_detection_ensemble.py) and [train_medical_imaging_ensemble.py](./train_medical_imaging_ensemble.py) with the respective command line arguments.

Afterwards, [eval_face_detection_ensemble.ipynb](./eval_face_detection_ensemble.ipynb) and [eval_medical_imaging_ensemble.ipynb](./eval_medical_imaging_ensemble.ipynb) have to be executed to evaluate all (tasks, runs, models) combinations.

Finally, the analyze_*.ipynb files can be used to execute the individual analyses.