# FURINA: Free from Unmergeable Router via LINear Aggregation of mixed experts

## Acknowledgement
This code base is modified based on MoE-PEFT code base. Please do not utilize this code base to reproduce other MoE-LoRA results, which may resulting in unexpected error.
We utilize the original MoE-PEFT framework for reproducing performance of other MoE--LoRA approaches.

## Installation

Please refer to [MoE-PEFT Install Guide](./Install.md)

## Direct training with bash
```bash
# Fine-tune llama3.2-3B on arc-c dataset, GPU_id=0, with a custom suffix `my_experiment`
bash ./training_bash.sh llama3.2-3B furina arc-c 0 my_experiment
```

## Merging the fine-tuned weights
```bash
python merge_m2loraSE.py
```
The adapter path and save-to path are set within the code.