# Efficient Inference With Model Cascades

Supplementary Material for the 2023 TMLR Paper Submission: Efficient Inference With Model Cascades

# Environment:
Environment requirements are provided in ```requirements.txt```.

The results were computed in a Conda enviroment with Python 3.10 created with the following commands (with exact versions commented):
```bash
conda create -n eec
conda activate eec
conda install pandas numpy matplotlib -c conda-forge
# pandas==1.4.4, numpy==1.23.2, matplotlib==3.5.3
conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c conda-forge
# torch==1.12.1, torchvision==0.13.1, torchaudio==0.12.1, cudatoolkit==11.6.0
conda install -c fvcore -c iopath -c conda-forge fvcore iopath
# fvcore==0.1.5.post20220512, iopath==0.1.9
pip install timm
# timm==0.6.7
pip install git+https://github.com/mapillary/inplace_abn.git@v1.0.12
# inplace-abn==1.0.12, to obtain MAC for TResNet, fails to install via requirements.txt and is not essential
pip install transformers
# transformers==4.21.3
pip install datasets
# datasets==2.4.0
pip install sentencepiece
# sentencepiece==0.1.97, for ALBERT
pip install protobuf==3.20.1 # for ALBERT
```

# Instructions:
The code is separated in mostly self-contained files to enable easy modification and partial execution. All scripts should be executed from the directory they are in.

All code for the ImageNet experiments is contained in the ```imagenet``` folder. It also contains stored Pareto fronts, but not cascades, logits, and inference data due to their large file size. The ImageNet validation set and ImageNetV2 MatchedFrequency test set are required with standard class subfolders structure. The ImageNet test set structure should be the default single folder containing all the unlabelled images. Follow these steps in order to reproduce the experiments:
1. Enter the paths to the required datasets in ```config.py``` and modify batch size and number of workers depending on the hardware you use.
2. ```python prepare_dataframe.py``` obtains a dataframe with information for all models and lists of Pareto optimal models.
3. ```python inference.py``` infers the ImageNet validation set for all Pareto optimal models and saves both logits and inference data for the cascades. Use -h to see supported options.
4. ```python paretos.py``` creates the baseline Pareto fronts and computes all 2-model cascade Pareto fronts for both MAC and time cost. Use -h to see supported options.

The remaining order is irrelevant.
- ```distribution_shift.py``` computes the distribution shifted ImageNet 2-model maximum softmax cascade Pareto front on ImageNetV2 as well as the ImageNetV2 maximum softmax cascade Pareto front. Use -h to see supported options.
- ```tri_cascades.py``` computes 3-model cascade Pareto fronts, which are used for Figure 4 and Figure 7. This may take around 1-3 days depending on your CPU. A cascading method needs to be specified with the -c flag. The script computes the Pareto front for one cascading method at a time. This can be parallelized by running the script multiple times simultaneously.
- ```cascade_evaluation.py``` creates a dataframe with evaluation data for individual cascades, which is used for Figure 2.
- ```appendix_top5.py``` computes 2-model cascade Pareto fronts for the trade-off between top-5 accuracy and MAC. They are used for Figure 9.
- ```test_submissions.py``` creates the ImageNet evaluation server submissions whose results are discussed in the Paper.

All code and generated data for the GLUE benchmark experiments are contained in the ```glue``` folder. Follow these steps in order to reproduce the experiments:
1. ```python inference.py``` performs inference for all models and datasets to obtain logits and inference data for the cascades.
2. ```python paretos.py``` computes all 2-model cascade Pareto fronts.

The files to recreate all figures are contained in the respective ```figures``` folders.