# Instructions on Repeating the Experiments
The full implementation of the hybrid-tuning is included in the `OurTrainer` class with setting `self.args.trainer = 'hybrid'` in `trainer.py`. By default, the trainer behavior is the same as the original implementation of ZO-Bench. 
## Repeating the Hybrid-Tuning for Specific PEFT Method and Base Model on a Given Task 
1. Clone the ZO-Bench repository
```bash
git clone https://github.com/ZO-Bench/ZO-LLM.git
```

2. Overwrite modified files using our provided source codes. For example, copy `trainer.py` to `ZO-LLM/zo-bench/trainer.py` to replace the original file.

3. Follow the `requirements.txt` to install the necessary dependencies. 
```bash
pip install -r requirements.txt
```

4. Run the experiments by executing the following command (a sample command for running one experiment with a set of given hyperparameters):
```bash
python3 ./zo-bench/run.py --hybrid_tuning --prompt_tuning --num_virtual_tokens=10 --prompt_init_by_real_tokens --model_name=meta-llama/Llama-2-7b-hf --task_name=SST2 --num_train_epochs=5 --no_reparam --per_device_train_batch_size=16 --load_best_model_at_end --evaluation_strategy=steps --save_strategy=steps --save_total_limit=1 --eval_steps=1000 --max_steps=20000 --logging_steps=10 --num_eval=1000 --num_train=1000 --num_dev=100 --train_as_classification --perturbation_mode=two_side --trainer=hybrid --optimizer=sgd --train_set_seed=0 --lr_scheduler_type=constant --eval_steps=500 --save_steps=500 --save_model --learning_rate=1e-3 --hybrid_scale=1e-3 --weight_decay=0 --zo_eps=1e-5 
```

All hyperparameters are provided in our sweeps configurations. 

## Repeating the Hybrid-Smoothness Visualization 
1. Activate the same virtual environment as ZO-Bench. 
2. Run each file in `Smoothness-Estimation` to record the smoothness estimation. All results will recorded in Weights & Biases (WandB) platform.