### This file contains instructions for running the code to reproduce our results in the manuscript ###

1. To reproduce our results in Figure 3
    (1) Check t1.ipynb and t2.ipynb to create multi-label MNIST and Cifar10 datasets
    (2) Train models using default classifier:
            python train_param_eff.py --arch $arch --dataset_root $data_path
        Train models using parameter efficient ETF classifier:
            python train_param_eff.py --arch $arch --etf --fix_dim --dataset_root $data_path
            
        Note that $arch should be chosen between [resnet18, resnet50, vgg16, vgg19]
            and $data_path should be chosen between [<root_dir>/mnist_combine.pkl, <root_dir>/c10_combine.pkl]
    (3) After having the model, validate the NC statistics by running:
        For model with default classifier:
            python validate_nc.py --checkpoint_path saved/param/<dataset>_<arch>_etf_False_fixdim_False_lambh/ --arch $arch --dataset_root $data_path
        For model with ETF classifier:
            python validate_nc.py --checkpoint_path saved/param/<dataset>_<arch>_etf_True_fixdim_True/ --arch $arch --etf --fix_dim --dataset_root $data_path
        
        The NC statistics can be found in the saved pickle file (in the model checkpoint directory, with the name "info.pkl"), with 'nc1', 'nc2_w', 'nc3', 'angle_metric' for NC1, NC2, NC3, NCm, respectively.