# Training and Evaluation

We provide bash scripts in [scripts/](../scripts) for each prompting variant including OpenDAS, vision, language and independent V-L prompting.
Make sure to configure the dataset paths in environment variable `DATA` and run the commands from the main directory `multimodal-prompt-learning/`.
Below we provide training and evaluation instructions for OpenDAS. The same instructions applies for all other variants including *Vision (VPT), Language and independent V-L prompting*.


### Training time and compute
We train OpenDAS on each dataset with a batch size of 16 using a **single** NVIDIA A100 GPU.
Training MaPle for 5 epochs takes 10-12 hour for a single seed.
To ease reproduction of OpenDAS results, we have provided [training logs](https://drive.google.com/drive/folders/1EvuvgR8566bL0T7ucvAL3LFVwuUPMRas?usp=sharing) for all datasets. 

## OpenDAS

#### (1) Base-to-Novel class generalization setting
The default training settings are provided in config file at `configs/trainers/OpenDAS/vit_b16_c2_ep5_batch4_2ctx.yaml`. All hyper-parameters such as prompt length, prompt depth, etc., can be modified using this config file.

Below, we provide instructions to train OpenDAS on <DATASET>. 


```bash
# Other possible dataset values includes [caltech101, food101, dtd, ucf101, oxford_flowers, oxford_pets, fgvc_aircraft, stanford_cars, sun397, eurosat]

# seed=1
# trains and evaluates on base classes
bash scripts/opendas/base2new_train_ivlp.sh <DATASET> 1
# evaluates on novel classes
bash scripts/opendas/base2new_test_ivlp.sh <DATASET> 1

# seed=2
# trains and evaluates on base classes
bash scripts/opendas/base2new_train_ivlp.sh <DATASET> 2
# evaluates on novel classes
bash scripts/opendas/base2new_test_ivlp.sh <DATASET> 2

# seed=3
# trains and evaluates on base classes
bash scripts/opendas/base2new_train_ivlp.sh <DATASET> 3
# evaluates on novel classes
bash scripts/opendas/base2new_test_ivlp.sh <DATASET> 3
```

#### Averaging results over 3 seeds: 
Once the above trainings and evaluations are completed, the `output/` directory should have the following structure:

```
output
|–– base2new/
|   |–– test_new/
|   |   |–– <DATASET>/
|   |   |   |–– shots_16/
|   |   |   |   |–– OpenDAS/
|   |   |   |   |   |–– vit_b16_c2_ep5_batch4_2ctx/
|   |   |   |   |   |   |–– seed1/
|   |   |   |   |   |   |–– seed2/
|   |   |   |   |   |   |–– seed3/
|   |–– train_base/
|   |   |–– <DATASET>/
|   |   |   |–– shots_16/
|   |   |   |   |–– OpenDAS/
|   |   |   |   |   |–– vit_b16_c2_ep5_batch4_2ctx/
|   |   |   |   |   |   |–– seed1/
|   |   |   |   |   |   |–– seed2/
|   |   |   |   |   |   |–– seed3/
```

Now use the script `parse_test_res.py` and run the commands below to calculate the averaged results:
```bash
# prints averaged results for base classes
python parse_test_res.py output/base2new/train_base//<DATASET>/shots_16/OpenDAS/vit_b16_c2_ep5_batch4_2ctx
# averaged results for novel classes
python parse_test_res.py output/base2new/test_new/<DATASET>/shots_16/OpenDAS/vit_b16_c2_ep5_batch4_2ctx --test-log
```

The above steps can be repeated for other individual datasets.

#### Reproducing results using pre-trained weights for base-to-novel generalization setting

We show an example to reproduce results for <DATASET>. Follow the instructions below to reproduce results using our pre-trained model weights:
* Download the zipped folder containing pre-trained weights for a single dataset from this [link](https://drive.google.com/drive/folders/1-tB6BUDBzs9CXTOJ7p5hM4Svq1tL_mGz?usp=sharing). Additionally we also provide the log files for both training and evaluation. After unzipping, the directory should look like this:

```
<DATASET>
|–– base/
|   |–– seed1/
|   |–– seed2/
|   |–– seed3/
|–– novel/
|   |–– seed1/
|   |–– seed2/
|   |–– seed3/
```

Now use the evaluation script `scripts/opendas/reproduce_ivlp.sh` and run the commands below to calculate the averaged results:
```bash
# evaluate on base and novel classes for SEED1
bash scripts/opendas/reproduce_ivlp.sh <DATASET> 1 /path/to/<DATASET>/weights/folder
# evaluate on base and novel classes for SEED2
bash scripts/opendas/reproduce_ivlp.sh <DATASET> 2 /path/to/<DATASET>/weights/folder
# evaluate on base and novel classes for SEED3
bash scripts/opendas/reproduce_ivlp.sh <DATASET> 3 /path/to/<DATASET>/weights/folder
```

This should evaluate and save the log files in `output/` directory. To obtain the averaged results, run:

```bash
# prints averaged results for base classes
python parse_test_res.py output/base2new/train_base/<DATASET>/shots_16/OpenDAS/vit_b16_c2_ep5_batch4_2ctx
# averaged results for novel classes
python parse_test_res.py output/base2new/test_new/<DATASET>/shots_16/OpenDAS/vit_b16_c2_ep5_batch4_2ctx --test-log
```


#### (2) Cross-Dataset Transfer
We provide instructions to train OpenDAS on imageNet using all 1000 classes and then evaluating it directly on new downstream datasets.
We provide cross-dataset config for OpenDAS: `configs/OpenDAS/vit_b16_c2_ep5_batch4_2ctx_cross_datasets.yaml`.
* Firstly, train OpenDAS on <DATASET> in few-shot manner (for all 3 seeds).

```bash
# seed=1 
bash scripts/opendas/xd_train_ivlp.sh <DATASET> 1
# seed=2 
bash scripts/opendas/xd_train_ivlp.sh <DATASET> 2
# seed=3 
bash scripts/opendas/xd_train_ivlp.sh <DATASET> 3
```

* Now evaluate imageNet model on downstream datasets.

```bash
for SEED in 1 2 3
do
    bash scripts/opendas/xd_test_ivlp.sh caltech101 ${SEED}
    bash scripts/opendas/xd_test_ivlp.sh oxford_pets ${SEED}
    bash scripts/opendas/xd_test_ivlp.sh stanford_cars ${SEED}
done
```


You can obtain averaged results by using the script `parse_test_res.py` and following the similar steps as provided in base-to-novel generalization experiments.
<br>


#### Reproducing official results for cross-dataset and domain generalization setting

We provide the instructions below to reproduce domain-generalization and cross-datasets results using our pre-trained <DATASET> model weights for OpenDAS:
* Download the zipped folder containing pre-trained weights for <DATASET> from this [link](https://drive.google.com/drive/folders/1bmhvmNZc13WJ5U71qt0t8k91wyuoemVF?usp=sharing). Additionally, we also provide the log files for both training and evaluation. After unzipping, the directory should look like this:

```
<DATASET>
|–– seed1/
|–– seed2/
|–– seed3/
```

Now use the evaluation script `scripts/opendas/reproduce_ivlp_xd.sh` and run the commands below to calculate the averaged results:
```bash
# evaluate on given dataset for SEED1
bash scripts/opendas/reproduce_ivlp_xd.sh food101 1 /path/to/weights/folder
# evaluate on given dataset for SEED2
bash scripts/opendas/reproduce_ivlp_xd.sh food101 2 /path/to/weights/folder
# evaluate on given dataset for SEED3
bash scripts/opendas/reproduce_ivlp_xd.sh food101 3 /path/to/weights/folder
```

This should evaluate and save the log files in `output/` directory. To obtain the averaged results, run:

```bash
# prints averaged results for a dataset
python parse_test_res.py output/evaluation/OpenDAS/vit_b16_c2_ep5_batch4_2ctx_cross_datasets_16shots/<DATASET> --test-log
```


#### Training and Evaluating other variants

For other variants including vision, language and independent V-L prompting techniques, we provide their corresponding configs and scripts as follows.

```
configs
|–– datasets/
|–– trainers/
|   |–– CoCoOp/
|   |–– CoOp/
|   |–– OpenDAS/
|   |–– VPT/
|   |–– RPO/
```

```
scripts
|–– cocoop/
|–– coop/
|–– language-prompting/
|–– maple/
|–– rpo/
|–– opendas/
```

### Acknowledgement: These instructions are based on [MaPLe](https://github.com/muzairkhattak/multimodal-prompt-learning).

Please use the corresponding config and script files and follow the same instructions as provided for OpenDAS in order to train and evaluate the other variants. Same instructions can be followed to reproduce results of other variants using provided pretrained weights.
This repository also supports using official CoOp, CoCoOp, OpenDAS, RPO configs and models.
