## Getting Started
We started from the official implementation of **[Model Merging with SVD to Tie the KnOTS](https://arxiv.org/abs/2410.19735)**.
Vision datasets have to be downloaded on disk for this repo to work, while textual datasets are downloaded automatically.

### Installation
Create a virtual environment and install the dependencies:
```bash
conda env update --file conda_environment.yaml --prune
```
#### Evaluation
Any supported experimental suite can be run in one of our eval_scripts directory. 
All scripts record the performance for each model and write the results (+ other attributes) to a csv file.
- nli_pertask_linearsearch.py: This script conducts a linearsearch to find the best merging parameters for a specified experiment configuration in the NLI per-task setting.
- 8vision_pertask.py: This script evaluates a specified experiment configuration with set hyperparameters in the benchmark per-task setting.

#### Typical evaluation command
```bash
export PYTHONPATH=`pwd`; python -m eval_scripts.8vision_pertask --config_name vitB_r16_com
```