# MoSA

MoSA (Group-Share Adaptation) provides a parameter-sharing approach for efficiently fine-tuning large language models. This repository extends an open-source LLaMA fine-tuning framework with modules and examples focused on MoSA training.

## Features

- Parameter-efficient fine-tuning through Group-Share Adaptation.
- Compatible with the wide range of models supported by LLaMA Factory.
- Includes example scripts for training and evaluation with MoSA.

## Installation

Clone the repository and install the dependencies:

```bash
pip install -e .
pip install jieba
pip install nltk
pip install rouge_chinese
pip install tensorboard
pip install sacrebleu
```

## Quickstart

To launch a MoSA training run, use one of the provided example scripts:

```bash
bash examples/train_mosa/mosa.sh
```
The method implementation are shown in src/llamafactory/model/model_utils/mosa.py.
The script demonstrates how to configure the equivalent rank and target modules for MoSA.

## License

This project is licensed under the [Apache-2.0 License](LICENSE).

## Citation

Citation details will be provided after the review process to preserve anonymity.
