# Inner Ensemble Nets 

### Using IEN in your model 

IEN implementation is under ./src/iea.py
You can easily include it in your model and replace each nn.Conv2d with Conv2d_iea or nn.Linear with Linear_iea. 
All the functions have the exact interafce as Pytorch API, except parameter m which controls the number of inner ensembles. 
Also make sure not init your model after IEN initilization so it doesn't conflict. 

After training your model, you can use the weight mergeing techniques by doing executing this code: 
```python

for m in model.modules():
    if hasattr(m, "domms"):
        # print("Apply inv variance")
        m.domms = False
        m.apply_weights_pruning()
```



# Results Reproduction 


### VGG
For vgg, use the IEN/exps/vgg_cifar_10 folder to train and test models on cifar10. Similarly use IEN/exps/vgg_cifar_100 for cifar100. 

Use the following command line. (Only the important arguments are shown. For the other arguments, and the acceptable format of arguments, see the argument parser in main.py in the vgg folders)

python3 main.py -a <architecture> --save-dir <path to directory where checkpoints, etc. are saved> --version <you assign this number to prevent overwriting your checkpoints when multiple similar models are trained> --variation <base or ien or maxout or fc> --gpu <GPU index>

python3 main.py -a vgg16_bn --save-dir ~/IEN/output --version 3 --variation ien --gpu 0

To resume training from a checkpoint, add --resume <path to checkpoint file> flag.

After training, to get the evaluation results for the downsized models, rerun the command with the --prune flag.

Distributed training is possible but was not used for this paper. See https://github.com/pytorch/examples/tree/master/imagenet on how to format the commandline for distributed training.

For outer ensembles, use main_outer.py instead of main.py and add the --prune flag. This will currently need the 3 best checkpoints from versions 0, 1, and 2, of the same model, but can be changed for smaller or larger outer ensembles.

To change the value of m for IEN, maxout, and IEN+FC models, go to vgg_iea.py, vgg_maxout.py, and vgg_fc.py, respectively, and change every occurrence of X in m=X, to your desired value of m.
    
### ResNet 
Use the IEN/exps/resnet_cifar_10 folder to train and test models on cifar10. Similarly use IEN/exps/resnet_cifar_100 for cifar100. 

To execute a trainig task use: 
    
CUDA_VISIBLE_DEVICES=0 python -u trainer.py --version=1500   --arch=resnet110 --save-dir=./output/resnet_cifar_100/save_resnet110 --Mense=4 --model_type iea , the args are described in the training code. Nonetheless, the important ones are shown.

To test the models and obtain the merging results use execute test.py 
To test the mergin on base models use: testmerge.py 
To test the resuls of outer ensemble use: ensembleres.py.py
To obtain a csv file with all the results in a nice format use: results_check/checkResNetCF10Results.py

Please note all the previous 4 files have a path variable that points to the results directory, change this accordingly.
    
### DenseNet 
Use the IEN/exps/efficient_densenet_pytorch_cf10 folder to train and test models on cifar10. Similarly use IEN/exps/efficient_densenet_pytorch_cf100 for cifar100. 

To execute a trainig task use: 
    
CUDA_VISIBLE_DEVICES=0 python demo.py --data ./tds --depth 100 --growth_rate 12 --seed 1700 --model_type iea --Mense 4 --save ./output/save_normal --efficient True  , the args are described in the training code. Nonetheless, the important ones are shown.

To test the models and obtain the merging results use execute checkdensenet.py 
To test the mergin on base models use: checkmerge.py 
To test the resuls of outer ensemble use: checkensemble.py
To obtain a csv file with all the results in a nice format use: results_check/checkDenseNetCF10Results.py
Please note all the previous 4 files have a path variable that points to the results directory, change this accordingly.
    
### NAS Results 
We reused the original repo of NAS-Bench-201, we modified the cells operators and added ours + maxout ones. These can be found at: http://ip-172-31-68-43:8888/edit/IEN/exps/AutoDL-Projects/lib/models/cell_operations.py 
    
Then you can run the NAS serach algorithms using: 
CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/algos/GDAS.sh cifar10 1 -1
CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/algos/DARTS-V1.sh cifar10 1 -1
    
The you can train the discovered cells using: 
CUDA_VISIBLE_DEVICES=4,5,6,7 bash ./scripts-search/NAS-Bench-201/train-a-net.sh '|nor_conviea_3x3~0|+|nor_conviea_1x1~0|nor_conv_3x3~1|+|nor_conviea_1x1~0|nor_conviea_3x3~1|nor_conviea_3x3~2|' 16 5



###
All trained moders can be found here: http://s3.amazonaws.com/ienpretrainedmodel/ 
This will lis keys for al lthe files, for examples: 
<ListBucketResult>
<Name>ienpretrainedmodel</Name>
<Prefix/>
<Marker/>
<MaxKeys>1000</MaxKeys>
<IsTruncated>false</IsTruncated>
<Contents>
<Key>vgg_cifar100_trainedmodels/completion_script.sh</Key>
<LastModified>2020-06-11T10:24:22.000Z</LastModified>
<ETag>"6b65b6c3fee1ee34342c0a891ee0bc24"</ETag>
<Size>436</Size>
<StorageClass>STANDARD</StorageClass>
</Contents>

In order to download a specific file/trained model for instance vgg_cifar100_trainedmodels/completion_script.sh, select the key and construct this url:  http://s3.amazonaws.com/ienpretrainedmodel/vgg_cifar100_trainedmodels/completion_script.sh Then you will be able to download it.
    
The code is in a beta state, we will evolve it into a better structure. 
    
    
 
    

    
    