# GIFT: Generative Parameter-Efficient Fine-Tuning 

# Environment Setup
Please follow the instructions in [experiment_setup/SETUP.md](experiment_setup/SETUP.md) to setup the environment.

# Experiments and Setup
## Language Modeling
Please refer to [scripts/language_modeling/README.md](scripts/language_modeling/README.md) for details on data preparation and training for commonsense reasoning, arithmetic reasoning and instruction tuning experiments.

## Visual Recognition
### FGVC
Please refer to [scripts/visual_classification/fgvc/README.md](scripts/visual_classification/fgvc/README.md) for details on data preparation and training.

### VTAB
Please refer to [scripts/visual_classification/vtab/README.md](scripts/visual_classification/vtab/README.md) for details on data preparation and training.

## Natural Language Understanding using GLUE
Please refer to [scripts/language_understanding/README.md](scripts/language_understanding/README.md) for details on data preparation and training.

# Integration with HuggingFace PEFT

Our implementation is compatible with the HuggingFace PEFT interface.

```python
# Example for the beta configuration in Figure 1 in the paper
from transformers import AutoModelForCausalLM
from peft import get_peft_model, GIFTConfig

dtype = torch.bfloat16 # or torch.float32, if bfloat16 is not supported
backbone = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B",
    torch_dtype=dtype,
    device_map=<DEVICE>
)

target_modules = ["q_proj", "k_proj", "v_proj", "up_proj", "down_proj"]
# Modules to be tied are controlled through regex pattern
tied_modules = [
    ["model.layers.\d+.self_attn.q_proj"], 
    ["model.layers.\d+.self_attn.k_proj"], 
    ["model.layers.\d+.self_attn.v_proj"], 
    ["model.layers.\d+.mlp.up_proj"],
    ["model.layers.\d+.mlp.down_proj"]
]
# What dimension to apply GIFT to
transform_dim = {
    "q_proj": "input",
    "k_proj": "input",
    "v_proj": "input",
    "up_proj": "input",
    "down_proj": "input"
}
peft_config = GIFTConfig(
    target_modules=target_modules,
    tied_modules=tied_modules,
    transform_dim=transform_dim,
    rank=64,
    gift_alpha=64,
    task_type="CAUSAL_LM"
)

wrapped_model = get_peft_model(model, peft_config, adapter_name="gift")

# Now, you can train the model as you would normally do
# ...
# ...
```

```python
# Example for the delta configuration in Figure 1 in the paper
from transformers import AutoModelForCausalLM
from peft import get_peft_model, GIFTConfig

dtype = torch.bfloat16 # or torch.float32, if bfloat16 is not supported
backbone = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B",
    torch_dtype=dtype,
    device_map=<DEVICE>
)

target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "up_proj", "gate_proj", "down_proj"]
# Modules to be tied are controlled through regex pattern
tied_modules = []
for l in range(32): # Hardcoded for llama-7/8b for now.
    tied_modules += [
        [f"model.layers.{l}.self_attn.q_proj", f"model.layers.{l}.self_attn.k_proj", f"model.layers.{l}.self_attn.v_proj"],
        [f"model.layers.{l}.self_attn.o_proj"], 
        [f"model.layers.{l}.mlp.up_proj", f"model.layers.{l}.mlp.gate_proj"],
        [f"model.layers.{l}.mlp.down_proj"]
    ]
# What dimension to apply GIFT to
transform_dim = {
    "q_proj": "input",
    "k_proj": "input",
    "v_proj": "input",
    "o_proj": "output",
    "up_proj": "input",
    "gate_proj": "input",
    "down_proj": "output",
}
peft_config = GIFTConfig(
    target_modules=target_modules,
    tied_modules=tied_modules,
    transform_dim=transform_dim,
    rank=64,
    gift_alpha=64,
    task_type="CAUSAL_LM"
)

wrapped_model = get_peft_model(model, peft_config, adapter_name="gift")

# Now, you can train the model as you would normally do
# ...
# ...
```

# Acknowledgements
This code draws from [timm](https://github.com/huggingface/pytorch-image-models/tree/main), [TOAST](https://github.com/bfshi/TOAST), and [pyreft](https://github.com/stanfordnlp/pyreft), and [PEFT](https://github.com/huggingface/peft). We thank the authors for their amazing work.

