# GraphLog: A Benchmark suite for measuring Logical Generalization in Graph Neural Networks

an ICLR 2021 submission

## Repository organization

- Entrypoint: [codes/app/main.py](codes/app/main.py) for local and [codes/app/submitit_runner.py](codes/app/submitit_runner.py) for SLURM
- Config files : [config/](config/) directory
- Experiment files : [codes/experiment](codes/experiment/)
- Models : [codes/model](codes/model)
  - _Representation Functions_ : Param, GCN, GAT
  - _Composition Functions_ : RGCN, E-GAT

## Dataset

Dataset used in the paper [can be obtained here](https://drive.google.com/file/d/1nsVr-CXYouzrdiQUgqSbQLcfQJduTJyg/view?usp=sharing), and the generator scripts can be found in [lgw/](lgw/) folder.

## Training

Config files can be placed either directly in [config](config/) directory, or in any subfolder. For example, if `supervised.yaml` config file resides in [config/supervised](config/supervised) directory,
the way to invoke training is

```
python app/main.py --config_id supervised/supervised
```

## Evaluation

- [scripts/eval_supervised.py](scripts/eval_supervised.py) contains the file to run supervised / multitask / continual learning evaluations. Check [scripts/eval.sh](scripts/eval.sh) file for usage to evaluate individual tasks.
- Raw results used in the paper are available in [logs](logs/) folder

## Statistics and Results

- Statistics regarding all the worlds in **GraphLog** can be found in [notebooks/GraphLogStats.ipynb](notebooks/GraphLogStats.ipynb) and [notebooks/clean_data/graphlog_stats.csv](notebooks/clean_data/graphlog_stats.csv)
- Unprocessed and processed results can be found in [notebooks/raw_data/](notebooks/raw_data/) and [notebooks/clean_data/](notebooks/clean_data/) respectively.

