A Gradient Method for Multilevel Optimization
====

## Requirements
- Python 3.9.2
- As for requirements for libraries, see `requirements.txt`.

## Usage
### Reproducing the results in "5.1 Convergence to optimal solution"
1. Change the working directory to this folder.
```
cd codes
```
2. Run `three_points-exp.py`. To see the results with diffrent $(T_2, T_3)$, change the arguments `T_2` and `T_3` in `three_points-exp.py`.
```
python three_points-exp.py
```
3. The results will be stored in `result/three_points`. The time stamp at execution is used as the file name.
4. To reproduce the figures in the paper, execute the cells in `analysis-3pt.ipynb` by Jupyter Notebook.


### Reproducing the results in "5.2 Application to hyperparameter optimization"

#### Diabetes dataset
1. Change the working directory to this folder.
```
cd codes
```
2. Run `adv_dad_mlp-exp.py` and `adv_dad_mlp_2level-exp.py`.
```
python adv_dad_mlp-exp.py
python adv_dad_mlp_2level-exp.py
```
3. The results of trilevel optimization will be stored in `result/diabetes_mlp_dad` and the results of bilevel optimization will be stored in `result/diabetes_mlp_dad_2level` . The time stamp at execution is used as the file name.
4. To reproduce the figures in the paper, execute the cells in `adv_eval_mlp_diabetes.ipynb` by Jupyter Notebook.

#### Boston house-price dataset
1. Change the working directory to this folder.
```
cd codes
```
2. Run `adv_dad_mlp_boston.py` and `adv_dad_mlp_boston_2level.py`.
```
python adv_dad_mlp_boston-exp.py
python adv_dad_mlp_boston_2level-exp.py
```
3. The results of trilevel optimization will be stored in `result/boston_mlp_dad` and the results of bilevel optimization will be stored in `result/boston_mlp_dad_2level` . The time stamp at execution is used as the file name.
4. To reproduce the figures in the paper, execute the cells in `adv_eval_mlp_boston.ipynb` by Jupyter Notebook.

#### Red wine quality dataset
1. Change the working directory to this folder.
```
cd codes
```
2. Run `adv_dad_mlp_wine_red.py` and `adv_dad_mlp_wine_red_2level.py`.
```
python adv_dad_mlp_wine_red-exp.py
python adv_dad_mlp_wine_red_2level-exp.py
```
3. The results of trilevel optimization will be stored in `result/wine_red_mlp_dad` and the results of bilevel optimization will be stored in `result/wine_red_mlp_dad_2level` . The time stamp at execution is used as the file name.
4. To reproduce the figures in the paper, execute the cells in `adv_eval_mlp_wine_red.ipynb` by Jupyter Notebook.

#### White wine quality dataset
1. Change the working directory to this folder.
```
cd codes
```
2. Run `adv_dad_mlp_wine_white.py` and `adv_dad_mlp_wine_white_2level.py`.
```
python adv_dad_mlp_wine_white-exp.py
python adv_dad_mlp_wine_white_2level-exp.py
```
3. The results of trilevel optimization will be stored in `result/wine_white_mlp_dad` and the results of bilevel optimization will be stored in `result/wine_white_mlp_dad_2level` . The time stamp at execution is used as the file name.
4. To reproduce the figures in the paper, execute the cells in `adv_eval_mlp_wine_white.ipynb` by Jupyter Notebook.

#### Reproducing "Table 1: MSE by test data with Gaussian noise with standard deviation of 0.08, using early-stopped parameter for prediction."
The mean value and and the standard deviation in Table 1 are calculated in the last 4 cells in each notebook. The notebook name for each dataset is as follows:

- Diabetes dataset: `adv_eval_mlp_diabetes.ipynb`
- Boston house-price dataset: `adv_eval_mlp_boston.ipynb`
- Red wine quality dataset: `adv_eval_mlp_wine_red.ipynb`
- White wine quality dataset: `adv_eval_mlp_wine_white.ipynb`

In the last 4 cells in these notebooks, the following values are calculated:

- `noized_errors_stop_dad_mean[index]` indicates the mean value for trilevel optimiazaion models.
- `noized_errors_stop_dad_std[index]` indicates the standard deviation for trilevel optimiazaion models.
- `noized_errors_stop_2l_mean[index]` indicates the mean value for bilevel optimiazaion models.
- `noized_errors_stop_2l_std[index]` indicates the standard deviation for bilevel optimiazaion models.

### Reproducing the results in "Comparison to Algorithm 1 and an existing method [24]" in Appendix

1. Execute all the cells in `evolutionary_and_compare.ipynb` from top to bottom.
