# Auto-Ensemble Structure Learning of Large Gaussian Bayesian Networks

This directory contains the source code associated with the paper "Auto-Ensemble Structure Learning of Large Gaussian Bayesian Networks."

Due to extensive experiments conducted during our research, running all experiments can be time-consuming. To demonstrate reproducibility, we provide a step-by-step guide for reproducing our results on the alarm testing problems with 1000 variables.

**Note**: 

+ all metadata on the files in this directory has been cleaned up for **Anonymous**.
+ The results here might slightly differ from those in the paper due to variations in random seed generation. Nonetheless, the overarching **statistical comparisons and conclusions remain consistent**.
+ Our experiments utilized a server with a 96-core CPU. The algorithm inherently supports parallelism. Therefore, machines with limited parallelism might experience deviations in results due to parallelization bottlenecks.



**Table of Contents**

[TOC]

## Directory Structure

```tex
|-- __init__.py
|-- algorithms	# algorithm implementations
|   |-- __init__.py
|   |-- bpef_ae_runner.py
|   |-- bpef_fges_runner.py
|   |-- bpef_pcstable_runner.py
|   |-- fges_default_runner.py
|   |-- fges_params_runner.py
|   |-- fges_params_runner_bic.py
|   |-- pcstable_default_runner.py
|   |-- pcstable_params_runner.py
|   |-- pcstable_params_runner_bic.py
|   `-- utils
|       |-- tetrad-gui-7.2.2-launch.jar
|       `-- translate.py
|-- bpef_ae_testing.py	# testing SLE / SLE (Default) / SLE (Random)
|-- bpef_fges_testing.py	# testing PEF-fGES
|-- bpef_pcstable_testing.py	# testing PEF-PC-Stable
|-- configs	# configs file 
|   |-- alarm_evaluation.json
|   |-- default_ae_4.json
|   |-- fges_configs.json
|   |-- greedy_ae_4.json
|   |-- pcstable_configs.json
|   |-- random_ae_4.json
|   |-- random_config_generator.py
|   `-- training_set_sample.json
|-- datasets	# store datasets
|   |-- alarm_1000
|   |-- generate_training_problem_instances.py
|   |-- json2dataset.py
|   |-- original
|   |   |-- alarm.json
|   |   `-- alarm.rds
|   |-- rds2json.R
|   `-- training_set_sample
|-- fges_testing_parallel.py	# testing fGES 
|-- greedyae_trainer.py	# training SLE by Auto-SLE
|-- load_results.py	# load the testing result
|-- logs	# store the results of testing
|   |-- bpef_default_ae_alarm_1000
|   |   `-- result.json
|   |-- bpef_fges_alarm_1000
|   |   `-- result.json
|   |-- bpef_greedy_ae_alarm_1000
|   |   `-- result.json
|   |-- bpef_pcstable_alarm_1000
|   |   `-- result.json
|   |-- bpef_random_ae_alarm_1000
|   |   `-- result.json
|   |-- fges_alarm_1000
|   |   `-- result.json
|   `-- pcstable_alarm_1000
|       `-- result.json
|-- partitions	# partition utils used in pef
|   |-- __init__.py
|   |-- _nn_chain_improved.pyx
|   |-- partition_improved.py
|   `-- setup.py
|-- pcstable_testing.py	# testing PC-Stable
`-- smac_runner.py	# hyerparameter sampler used in Auto-SLE
```

## Dependencies

#### Hardware

+ Our experiments were conducted on a Linux server powered by an Intel(R) Xeon(R) Gold 6336Y CPU @ 2.40GHz with 96 cores and 768GB RAM, running Ubuntu 22.04.2 LTS.

#### Software

To execute the code, you'll require Python (using Conda for package management is recommended), R, C++, and Java.

+ Python packages:

```shell
conda create -n autosle python=3.11
conda activate autosle

conda install gxx_linux-64 gcc_linux-64 swig

pip install scipy
pip install scikit-learn
pip install networkx
pip install smac
pip install jpype1
pip install Cython
```

+ R packages:

```shell
install.packages("bnlearn")
install.packages("rjson")
install.packages("argparse")
```

+ For Java, it's recommended to use `amazon-corretto-17-x64-linux-jdk`. Also, remember to modify the JVM path in the following files:

```tex
|-- algorithms
|   |-- bpef_ae_runner.py
|   |-- bpef_fges_runner.py
|   |-- bpef_pcstable_runner.py
|   |-- fges_default_runner.py
|   |-- fges_params_runner.py
|   |-- fges_params_runner_bic.py
|   |-- pcstable_default_runner.py
|   |-- pcstable_params_runner.py
|   |-- pcstable_params_runner_bic.py
|   `-- utils
|       `-- translate.py
```

+ Additionally, compile the .pyx files in the `./partitions` folder using:
  + command: `python setup.py build_ext --inplace`


## Scripts for reproducing our experiments

Since running all the experiments would be quite long, below we provide an exmaple for reproducing our results on the alarm testing problems with 1000 variables.

#### Generate Problem Instances

+ Navigate to the `datasets` folder.
  +  `cd datasets`
  
+ Download the Alarm network structure (in .rds format) from the bnlearn repository.
  + https://www.bnlearn.com/bnrepository/alarm/alarm.rds
  + The downloaded *alarm.rds* file is already in the folder *datasets/original*
  
+ Convert the .rds file to .json format.
  + `Rscript rds2json.R --rds ./original/alarm.rds --json ./original/alarm.json`
  + The output *alarm.json* file is already in the folder *datasets/original*

+ Generate problem instances based on the base network structure.

  + ```shell
    mkdir alarm_1000
    
    python json2dataset.py --json ./original/alarm.json --k 28 --c 0.1 --n_samples 1000 --seed 1 --nx ./alarm_1000/alarm_1.json --npz ./alarm_1000/alarm_1.npz --npy ./alarm_1000/alarm_1.npy
    python json2dataset.py --json ./original/alarm.json --k 28 --c 0.1 --n_samples 1000 --seed 2 --nx ./alarm_1000/alarm_2.json --npz ./alarm_1000/alarm_2.npz --npy ./alarm_1000/alarm_2.npy
    python json2dataset.py --json ./original/alarm.json --k 28 --c 0.1 --n_samples 1000 --seed 3 --nx ./alarm_1000/alarm_3.json --npz ./alarm_1000/alarm_3.npz --npy ./alarm_1000/alarm_3.npy
    python json2dataset.py --json ./original/alarm.json --k 28 --c 0.1 --n_samples 1000 --seed 4 --nx ./alarm_1000/alarm_4.json --npz ./alarm_1000/alarm_4.npz --npy ./alarm_1000/alarm_4.npy
    python json2dataset.py --json ./original/alarm.json --k 28 --c 0.1 --n_samples 1000 --seed 5 --nx ./alarm_1000/alarm_5.json --npz ./alarm_1000/alarm_5.npz --npy ./alarm_1000/alarm_5.npy
    python json2dataset.py --json ./original/alarm.json --k 28 --c 0.1 --n_samples 1000 --seed 6 --nx ./alarm_1000/alarm_6.json --npz ./alarm_1000/alarm_6.npz --npy ./alarm_1000/alarm_6.npy
    python json2dataset.py --json ./original/alarm.json --k 28 --c 0.1 --n_samples 1000 --seed 7 --nx ./alarm_1000/alarm_7.json --npz ./alarm_1000/alarm_7.npz --npy ./alarm_1000/alarm_7.npy
    python json2dataset.py --json ./original/alarm.json --k 28 --c 0.1 --n_samples 1000 --seed 8 --nx ./alarm_1000/alarm_8.json --npz ./alarm_1000/alarm_8.npz --npy ./alarm_1000/alarm_8.npy
    python json2dataset.py --json ./original/alarm.json --k 28 --c 0.1 --n_samples 1000 --seed 9 --nx ./alarm_1000/alarm_9.json --npz ./alarm_1000/alarm_9.npz --npy ./alarm_1000/alarm_9.npy
    python json2dataset.py --json ./original/alarm.json --k 28 --c 0.1 --n_samples 1000 --seed 10 --nx ./alarm_1000/alarm_10.json --npz ./alarm_1000/alarm_10.npz --npy ./alarm_1000/alarm_10.npy
    ```

  + for Alarm with *37* nodes, when *k=28*，the final problem instances with *37\*28=1036* nodes

+ Now, we get 10 problem instances of 1,000 nodes based on Alarm base network.

#### Constructing SLE by Auto-SLE

We've provided the SLE config file generated by Auto-SLE due to its time-intensive nature. 

Find it at `./configs/greedy_ae_4.json`. 

If you wish to proceed end-to-end, follow these steps:

+ Generate training data instances.
  + use *./datasets/generate_training_problem_instances.py* generate the training data instances (this processes is randomly, each time you can get different tranning set) (before generating, you need to downlad all base structure from *bnlearn*)

+ Construct the SLE.
  + use *./greedyae_trainer.py* contruct the SLE (finally, the SLE configs file will in the output directory), the hyperparameters space config in the *./configs/pcstable_configs.json* and *./configs/fges_configs.json*
  + a example (very small train set): `python greedyae_trainer.py --train ./configs/training_set_sample.json --output temp` (Update: the data of training_set_sample had been deleted because of the oversize of the supplementary material)

#### SLE (Default) and SLE (Random)

Generate SLE configurations within the `configs` folder.

+  `cd configs`

For SLE (Default)

+ `python random_config_generator.py --with_default True --output default_ae_4.json`

For SLE (Random)

+ `python random_config_generator.py --with_default False --output random_ae_4.json`

#### Testing

Run the following commands to execute all algorithms:

```shell
# PEF-SLE
nohup python bpef_ae_testing.py --datasets ./configs/alarm_evaluation.json --pap ./configs/greedy_ae_4.json --parallel 1 --output ./logs/bpef_greedy_ae_alarm_1000 > ./logs/bpef_greedy_ae_alarm_1000.log 2>&1 &

# PEF-SLE (Default)
nohup python bpef_ae_testing.py --datasets ./configs/alarm_evaluation.json --pap ./configs/default_ae_4.json --parallel 1 --output ./logs/bpef_default_ae_alarm_1000 > ./logs/bpef_default_ae_alarm_1000.log 2>&1 &

# PEF-SLE (Random)
nohup python bpef_ae_testing.py --datasets ./configs/alarm_evaluation.json --pap ./configs/random_ae_4.json --parallel 1 --output ./logs/bpef_random_ae_alarm_1000 > ./logs/bpef_random_ae_alarm_1000.log 2>&1 &

# PEF-PC-Stable
nohup python bpef_pcstable_testing.py --datasets ./configs/alarm_evaluation.json --parallel 1 --output ./logs/bpef_pcstable_alarm_1000 > ./logs/bpef_pcstable_alarm_1000.log 2>&1 &

# PEF-fGES
nohup python bpef_fges_testing.py --datasets ./configs/alarm_evaluation.json --parallel 1 --output ./logs/bpef_fges_alarm_1000 > ./logs/bpef_fges_alarm_1000.log 2>&1 &

# PC-Stable
nohup python pcstable_testing.py --datasets ./configs/alarm_evaluation.json --parallel 1 --output ./logs/pcstable_alarm_1000 > ./logs/pcstable_alarm_1000.log 2>&1 &

# fGES
nohup python fges_testing_parallel.py --datasets ./configs/alarm_evaluation.json --parallel 1 --output ./logs/fges_alarm_1000 > ./logs/fges_alarm_1000.log 2>&1 &
```

#### Results Analysis

+ load results by `python load_results.py --result xxx/result.josn`
+ PEF-SLE compare to base algorithms — PC-Stable and fGES
  + according to further statistical tests, **PEF-SLE significant better**

```shell
python load_results.py --result ./logs/bpef_greedy_ae_alarm_1000/result.json 
# F1 Adjacent: 0.812±0.014
# F1 Arrowhead: 0.678±0.034
# Runtime: 8.5±0.3

python load_results.py --result ./logs/pcstable_alarm_1000/result.json 
# F1 Adjacent: 0.724±0.007
# F1 Arrowhead: 0.503±0.005
# Runtime: 304.1±42.2

python load_results.py --result ./logs/fges_alarm_1000/result.json
# F1 Adjacent: 0.572±0.005
# F1 Arrowhead: 0.492±0.016
# Runtime: 657.6±23.0
```

+ Ablation study of Auto-SLE：comparing PEF-SLE、PEF-SLE (Default)、PEF-SLE (Random)、PEF-PC-Stable & PEF-fGES
  + According to further statistical tests, **PEF-SLE significant better in accuracy (F1 score)**

```shell
python load_results.py --result ./logs/bpef_greedy_ae_alarm_1000/result.json 
# F1 Adjacent: 0.812±0.014
# F1 Arrowhead: 0.678±0.034
# Runtime: 8.5±0.3

python load_results.py --result ./logs/bpef_default_ae_alarm_1000/result.json 
# F1 Adjacent: 0.677±0.014
# F1 Arrowhead: 0.567±0.031
# Runtime: 8.5±0.3

python load_results.py --result ./logs/bpef_random_ae_alarm_1000/result.json
# F1 Adjacent: 0.719±0.019
# F1 Arrowhead: 0.423±0.018
# Runtime: 6.1±0.1

python load_results.py --result ./logs/bpef_default_ae_alarm_1000/result.json 
# F1 Adjacent: 0.755±0.017
# F1 Arrowhead: 0.478±0.017
# Runtime: 3.2±0.2

python load_results.py --result ./logs/bpef_random_ae_alarm_1000/result.json
# F1 Adjacent: 0.677±0.014
# F1 Arrowhead: 0.567±0.031
# Runtime: 4.8±0.3
```