<!---
Copyright 2021 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

# Question answering

This folder contains several scripts that showcase how to fine-tune a 🤗 Transformers model on a question answering dataset,
like SQuAD. 

## Setup
```
pip install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html
pip install -r examples/summarization/requirements.txt 
pip install -e . 
pip install -e loralib/
export MASTER_PORT=8679
```

## Command for SQuADv2

Demo command for LoRA on DebertaV3-base: 

```
python -m torch.distributed.launch --master_port=$MASTER_PORT --nproc_per_node=1 examples/question-answering/run_qa.py --model_name_or_path microsoft/deberta-v3-base --dataset_name squad_v2 --apply_lora --lora_type frd --target_rank 1   --lora_r 1   --select_metric iptAB,magE,sumAB,onlyE --reg_orth_coef 0.0 --init_warmup 800000 --final_warmup 25000 --mask_interval 300 --finalize_rank --beta1 0.85 --beta2 0.00 --lora_module query,key,value,intermediate,layer.output,attention.output --lora_alpha 16 --do_train --do_eval --version_2_with_negative --max_seq_length 384 --doc_stride 128 --per_device_train_batch_size 16 --learning_rate 8e-5 --num_train_epochs 8 --warmup_steps 1000 --per_device_eval_batch_size 128 --evaluation_strategy steps --eval_steps 3000 --save_strategy steps --save_steps 100000 --logging_steps 300 --tb_writter_loginterval 300 --report_to tensorboard --seed 9 --root_output_dir ./log --sub_output_dir run --overwrite_output_dir 
```

Rankselection for DebertaV3-base: 
```
python -m torch.distributed.launch --master_port=$MASTER_PORT --nproc_per_node=1 examples/question-answering/run_qa.py --model_name_or_path microsoft/deberta-v3-base --dataset_name squad_v2 --apply_lora --apply_rankselector --lora_type svd --target_rank 1   --lora_r 2   --select_metric iptAB,iptE,sumAB,sumE --reg_orth_coef 0.1 --init_warmup 5000 --final_warmup 50000 --mask_interval 100 --finalize_rank --beta1 0.85 --beta2 0.85 --lora_module query,key,value,intermediate,layer.output,attention.output --lora_alpha 16 --do_train --do_eval --version_2_with_negative --max_seq_length 384 --doc_stride 128 --per_device_train_batch_size 16 --learning_rate 1e-3 --num_train_epochs 12 --warmup_steps 1000 --per_device_eval_batch_size 128 --evaluation_strategy steps --eval_steps 3000 --save_strategy steps --save_steps 100000 --logging_steps 300 --tb_writter_loginterval 300 --report_to tensorboard --seed 9 --root_output_dir ./log --sub_output_dir run --overwrite_output_dir 
```

