This is the code for PF2LoRA on question answering task 
### Abstract
We study the task of personalized federated fine-tuning with heterogeneous data in the context of language models, where clients collaboratively fine-tune a language model (e.g., BERT, GPT) without sharing their local data, achieving personalization simultaneously. While recent efforts have applied parameter-efficient fine-tuning techniques like low-rank adaptation (LoRA) in federated settings, they typically use single or multiple independent low-rank adapters with predefined maximal and minimal ranks, which may not be optimal for diverse data sources over clients.

To address this issue, we propose PF2LoRA, a new personalized federated fine-tuning algorithm built on a novel \emph{automatic rank learning approach via two-level LoRA}. Given the pretrained language model whose weight is frozen, our algorithm aims to learn two levels of adaptation simultaneously: the first level aims to learn a common adapter for all clients, while the second level fosters individual client personalization. A key advantage of PF2LoRA is its ability to adaptively determine a suitable rank based on an individual client’s data, rather than relying on a predefined rank that is agnostic to data heterogeneity. We present a synthetic example that highlights how PF2LoRA automatically learns the ground-truth rank for each client, tailoring the adaptation to match the properties of their individual data. Notably, this approach introduces minimal additional memory overhead, as the second-level adaptation comprises a small number of parameters compared to the first level. Our experiments on natural language understanding and generation tasks demonstrate that PF2LoRA significantly outperforms existing federated fine-tuning methods.

### Implementation
Our personalized federatad fine-tuning experiment is run on 8 NVIDIA A100 GPU, run the experiment by:

`bash run.sh`

There are several important parameters:
`dataset`: the name of dataset [squad, squad_v2]
`name_or_path`: the name or path of the pre-trained model 
`WORKERS`: world size 
`NUM_GPUS`: the number of gpus
`m`: method name [pf2lora, hetlora]
`rank_mat`: the lora rank of matrix
`rank_max`: r_max for hetlora
`rank_min`:  r_min for hetlora
`lr_B`: the learning rate for the common adapter
`lr_A`: the learning rate for the client-specific adapter
`gamma`: the sparsity parameter for hetlora 
`lamb`: the regularization factor
`com_rounds`: the total communication rounds
`com_interval`: the communication interval
`heteroogeneity` the heteroogeneity level