# Synergistic-conflict Effect Aware Merging (SEAM)

This repository is the official implementation of [Beyond Conflict: The Role of Synergistic Effects in Model Merging](https://arxiv.org/abs/2030.12345). 

![Overview](Overview.jpg)

## Requirements

To install requirements:

```setup
conda env create -f environment.yml
```

## Datasets
Refer to dataset processing in the [task_vectors](https://github.com/mlfoundations/task_vectors).

Or you can download the processed data from prior work [AdaMerging](https://github.com/EnnengYang/AdaMerging/tree/main).

Then modify `args.data_location` in main.py to the corresponding folder.

## Checkpoints

You can download the fine-tuned checkpoints from the [task_vectors#checkpoints](https://github.com/mlfoundations/task_vectors#checkpoints).
The Google Drive folder is: [task_vectors_checkpoints](https://drive.google.com/drive/folders/1u_Tva6x0p6oxu5Eo0ZZsf-520Cc_3MKw)

Then modify `args.save` in main.py to the corresponding folder.

*Note: When using ```torch.load(xxx_checkpoint).state_dict()``` fails, you can try ```pickle.load(open(xxx_checkpoint, 'rb')).state_dict()```.*


## Evaluation

To evaluate SEAM-B on eight-merging tasks, run:

```eval
python main.py --merge SEAM-B --alpha 1.8 --beta 0.5
```

To evaluate SEAM-O on eight-merging tasks, run:

```eval
python main.py --merge SEAM-O
```



## Acknowledgement

Our implementation references the code below, thanks to them.

- Task Arithmetic: https://github.com/mlfoundations/task_vectors

- TIES-MERGING: https://github.com/prateeky2806/ties-merging/tree/main

- DARE: https://github.com/yule-BUAA/MergeLM

- AdaMerging: https://github.com/EnnengYang/AdaMerging/blob/main
  
- TALL Masks: https://github.com/nik-dim/tall_masks

- TSV-M: https://github.com/AntoAndGar/task_singular_vectors
