# HELLORA: HOT EXPERTS LAYER-LEVEL LOW-RANK ADAPTATION FOR MOE MODEL
## Installation

Create and activate a Conda environment:
```
conda create -n lori python=3.10 -y
conda activate lori
```

```
pip install -r requirements.txt
```

### Example script

Training code generation adapters `HeLLoRA` on the gsm8k dataset using Olmoe-1b-7b with rank 32:

```
bash scripts/gsm8k_olmoe_r_32.sh
```

Feel free to reach out if you have any questions!

