# CA-LoRA-T5

The experiment code is modified based on https://github.com/OpenBMB/ModelCenter.

## Data Setup

Download SuperGLUE datasets from https://dl.fbaipublicfiles.com/glue/superglue/data/v2/combined.zip, unzip the package and rename the folder as `superglue`.

Then place `superglue` folder under `calora/down_data` folder. After this step, the file structure should be 
```
calora-T5
    - down_data
        - superglue
            - BoolQ
            - CB
            - ...
    - examples
    - modelcenter
    - ...
```

## Prepare Environment
Run the following script.
```
bash examples/t5/prepare.sh
```

## Run experiments

All testing scripts are in the directory ``examples/t5/scripts_*``. Select the script you want and run it. 

Take evaluate CA-LoRA on mixed compressed T5 on BoolQ as an example: the script is ``examples/t5/BoolQ-x-rd-rc-d.sh``. Run the following command.

```
bash examples/t5/BoolQ-x-rd-rc-d.sh
```

Useful abbreviations:
- q: quantization
- p: unstructured pruning
- s: structured pruning
- m: moefication
- x: mixed compression
- i: with inherit on
- rd: with inherit off
- rc: with recovery module
- d: with distill on

## Command line parameters

While launching the python program of our CA-LoRA, hyperparameters and some options are set by command line parameters. You can change them to test different configurations. In README, we introduce the meanings of some important command line parameters.

```
pet: ['True', 'False'], whether to use parameter efficient tuning (LoRA) or tune all parameters of compressed model
comp-type: ['quant', 'moe', 'pr', 'spr', 'mix', 'none'], method of model compression
pet-init-type: ['random', 'inherit'], method to initianlize PET modules
recover: ['True', 'False'], whether to add recovery module
distill: ['True', 'False'], whether to add distillation loss
quant-ckpt-path: str, checkpoint path of quantized model
moe-ckpt-path: str, checkpoint path of moefication model
pr-ckpt-path: str, checkpoint path of unstructure pruned model
spr-ckpt-path: str, checkpoint path of structure pruned model
model-ckpt-path: str, checkpoint path of pretrained model
mix-ckpt-path: str, checkpoint path of mix-compressed model
inherit-ckpt-path: str, checkpoint path of LLM with PET (only PET parameters are used)
quant-config-path: str, config path of quantization
pr-config-path: str, checkpoint path of unstructure pruning
spr-config-path: str, checkpoint path of structure pruning
```

