# A Hardware-Aware Framework for Accelerating Neural Architecture Search Across Modalities (AutoML Conference 2022)

The repository includes files neccessary for running the MobileNetV3 experiments from the manuscript: "A Hardware-Aware Framework for Accelerating Neural Architecture Search Across Modalities."

# Setup

## Environment

Our experiments were performed on various CPU and GPU hardware platforms. This repository provides an example which offers an easy way to reproduce our results given following requirements are met:

1. OS: Ubuntu 20.04, or similar
2. Python libraries, as listed in the `requirements.txt`, are installed.
* `pip install numpy==1.19.0`
* `pip install -r requirements.txt`
3. Download and install the imagenet-ilsvrc2012 dataset: https://image-net.org/download.php
4. `NVIDIA CUDA >= 11.0` is installed. (if using GPU for measuring top-1 accuracy)
5. System has `CUDA >= 11.0` compatible GPU available. (if using GPU for measuring top-1 accuracy)

Note that for a quick demonstration of the evolutionary algorithms and LINAS approaches, we provide a proxy top-1 accuracy predictor that will offset the need for top-1 validation measurements using a GPU (steps 4-5 above) and is enabled by NOT including the `--acc_validate` option in the run commands below.

## MobileNetV3 Super-network
This repository currently supports the Once-for-all (OFA) MobileNetV3 pre-trained super-network with a search space of 10^19. Download the OFA MobileNetV3-w1.0 `ofa_mbv3_d234_e346_k357_w1.0` pretrained model and place in the `collateral` folder:

* https://github.com/mit-han-lab/once-for-all
* https://drive.google.com/drive/folders/10leLmIiMtaRu4J46KwrBaMydvQt0qFuI?usp=sharing

# Running Experiments

In our work, we directly measure latency on various CPU and GPU hardware platforms. Because latency are very dependent on specific hardware settings and background processes, we provide a stage-wise latency look-up-table (LUT) for reproducibility. The `collateral/latency_lut_a100_b128` provides the stage-wise latency data as measured on a NVIDIA A100 GPU with a batch size of 128. Next, the `--dataset_path` will need to be updated to your local copy of the imagenet-ilsvrc2012 dataset. We use 5 random seeds of range [0, 1, 2, 3, 4] for the experiments below. Raw data for the below predictor-based experiments is located in the `results_raw` path.

## Evolutionary algorithm comparison

For running the evolutionary algorithm comparison (Figure 4 from the paper) please run the following script. Note that significant compute resource is required to validate each sub-network since there are a lot of evaluations in this example. We recommend using the accuracy predictor (first option) for a faster and more energy efficient demonstration and comparison of evolutionary algorithms for the sub-network search task.

* `python run_evoalgo_comparison.py --plot_results` (top-1 accuracy predicted - runs fast)
* `python run_evoalgo_comparison.py --plot_results --acc_validate` (top-1 accuracy validated - requires GPU and significant compute time)

Simulation results will be in the `results` path as .csv files with a plot as `hypervolume_ea_tournament.png`. Following these instructions should result in the follow figure being created that compares the various evolutionary algorithms in the MobileNetV3 top-1 accuracy and latency search space.

![Hypervolume comparison for various EA methods](./results_raw/hypervolume_ea_tournament.png)

## LINAS vs. NSGA-II vs. Random search comparison

For running the LINAS algorithm comparison (Figure 5 and 6 from the paper) please run the following script.

`python run_linas_comparison.py --plot_results --acc_validate` (top-1 accuracy validated - recommended)

Note that if a GPU is unavailable, the following can be run which treats a pre-trained accuracy predictor as the top-1 accuracy ground truth.

`python run_linas_comparison.py --plot_results` (top-1 accuracy predicted)

Simulation results and plots will be in the `results` path as .csv files.

![Scatterplot for LINAS and NSGA-II](./results_raw/scatterplot_linas_nsga2_random.png)

![Hypervolume comparison for LINAS and NSGA-II](./results_raw/hypervolume_linas_comparison.png)

# LICENSE

The code in this work is licensed under the Apache License 2.0.

Additionally, the most leveraged libraries in the repository have the following licenses:
* [pymoo](https://github.com/anyoptimization/pymoo) (Apache License 2.0)
* [Once-for-all](https://github.com/mit-han-lab/once-for-all) (MIT License)