# Conditional Execution Of Cascaded Models Improves The Accuracy-Efficiency Trade-Off

Supplementary Material for the ICLR 2023 Paper Submission: Conditional Execution Of Cascaded Models Improves The Accuracy-Efficiency Trade-Off

# Environment:
Environment requirements are provided in ```requirements.txt```.

The results were computed in a conda enviroment created with the following commands (with exact versions commented):
```bash
conda create -n eec
conda activate eec
conda install pandas numpy matplotlib -c conda-forge
# pandas==1.4.4, numpy==1.23.2, matplotlib==3.5.3
conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c conda-forge
# torch==1.12.1, torchvision==0.13.1, torchaudio==0.12.1, cudatoolkit==11.6.0
conda install -c fvcore -c iopath -c conda-forge fvcore iopath
# fvcore==0.1.5.post20220512, iopath==0.1.9
pip install timm
# timm==0.6.7
pip install git+https://github.com/mapillary/inplace_abn.git@v1.0.12
# inplace-abn==1.0.12, to obtain MAC for TResNet, fails to install via requirements.txt and is not essential
pip install transformers
# transformers==4.21.3
pip install datasets
# datasets==2.4.0
pip install sentencepiece
# sentencepiece==0.1.97, for ALBERT
pip install protobuf==3.20.1 # for ALBERT
```

# Instructions:
The code is separated in mostly self-contained files for easy modification and partial execution. All scripts should be executed from the directory they are in.

All files for the ImageNet experiments are contained in the ```imagenet``` folder. Follow these steps in order to reproduce the experiments:
1. Enter the paths to the required datasets in ```config.py``` and modify batch size and number of workers depending on the hardware you use.
2. ```python prepare_dataframe.py``` obtains a dataframe with information for all models and lists of Pareto optimal models.
3. ```python inference.py``` infers the ImageNet validation set for all Pareto optimal models and saves both logits and inference data for the cascades. Use -h to see supported options.
4. ```python paretos.py``` creates the baseline Pareto fronts and computes all 2-model cascade Pareto fronts for both MAC and time cost. Use -h to see supported options.

The remaining order is irrelevant.
- ```tri_cascades.py``` computes 3-model cascade Pareto fronts, which are used for Figure 4 and Figure 7. This may take around 1-3 days depending on your CPU. A cascading method needs to be specified with the -c flag. The script computes the Pareto front for one cascading method at a time. This can be parallelized by running the script multiple times simultaneously.
- ```cascade_evaluation.py``` creates a dataframe with evaluation data for individual cascades, which is used for Figure 2.
- ```appendix_top5.py``` computes 2-model cascade Pareto fronts for the trade-off between top-5 accuracy and MAC. They are used for Figure 9.
- ```test_submissions.py``` creates the ImageNet evaluation server submissions whose results are discussed in the Paper.

All files for the GLUE benchmark experiments are contained in the ```glue``` folder. Follow these steps in order to reproduce the experiments:
1. ```python inference.py``` performs inference for all models and datasets to obtain logits and inference data for the cascades.
2. ```python paretos.py``` computes all 2-model cascade Pareto fronts.

The files to recreate all figures are contained in the respective ```figures``` folders.