<!---
Copyright 2020 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

## Summarization

### Setup

In the dir `QA_Summarization`, run 
```
pip install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html
pip install -r examples/summarization/requirements.txt 
pip install -e . 
pip install -e loralib/
```

### Run

The example command of runing rank selection with Bart-large on XSUM: 
```
accelerate launch --multi_gpu --num_machine=1 --num_processes=8 --main_process_port=8679 --mixed_precision="no" \
examples/summarization/run_summarization_no_trainer.py \
--model_name_or_path facebook/bart-large \
--dataset_name xsum \
--apply_lora --apply_rankselector \
--lora_type svd --target_rank 1 --lora_r 2 --lora_alpha 32 \
--reg_orth_coef 0.1 \
--select_metric iptAB,iptE,sumAB,sumE \
--init_warmup 6000 --final_warmup 25000 --mask_interval 100 \
--beta1 0.85 --beta2 0.85 --finalize_rank \
--lora_module q_proj,k_proj,v_proj,out_proj,fc1,fc2 \
--per_device_train_batch_size 8 --learning_rate 5e-4 --num_train_epochs 15 \
--num_warmup_steps 3000 \
--max_source_length 768 --max_target_length 64 --max_length 768 \
--pad_to_max_length --num_beams 8 \
--per_device_eval_batch_size 8 --seed 9 \
--with_tracking --tb_writter_loginterval 500 \
--root_output_dir ./log \
--output_dir curve 
```


Comments: 
+ `apply_lora`: whether to apply lora. 
+ `apply_rankselector`: whether to apply MARVEL. 
+ `lora_type`: select formulation for $\Delta W$. Set `frd` for LoRA for full-rank decomposition. Set `svd` for MARVEL for SVD decompotion. 
+ `target_rank`: the final average target rank. 
+ `lora_r`: initial average rank. 
+ `reg_orth_coef`: orthongonal regularization weight for SVD formulation. 
+ `select_metric`: fixed as `iptAB,iptE,sumAB,sumE` for MARVEL. 
+ `init_warmup`: initial warmup for MARVEL. 
+ `final_warmup`: final warmup for MARVEL. 
+ `mask_interval`: masking interval for MARVEL. 
+ `beta1` and `beta2`: exponential moving average weight for sensitivity and uncertainty. 
+ `finalize_rank`: whether to finalize rank pattern during final warmup. 
+ `lora_module`: the modules to apply low rank adaptation. 
+ `root_output_dir`: the root dir to save all trials. 
+ `output_dir`: the postfix of output dir. 




