# Supplementary Material for AKBC submission

The following paragraphs explain how to reproduce the experiments in the paper and provide detailed information about the training of KGE models and hyperparameter searches performed.

## Reproducing results for one model/dataset

Reproducing the experiments is shown exemplary for the ComplEx model on the dataset CoDEx-M, e.g, the entry with an MRR equal to 0.349 in the table in Appendix A and the corresponding marker of Figure 1 in the main text.

To reproduce the example for ComplEx on CoDEx-M, please follow the steps below:

#### 1. Install the libKGE library in your python environment

Follow the quick start installation instructions of the libKGE library. Also run `download.sh` as mentioned in the installation instructions. To check if the installation was successful run `kge` in your console, which should output
`usage: kge [-h] {start,create,resume,eval,valid,test,dump,package} ...
kge: error: the following arguments are required: command`
Additionally keep track of the folder you cloned from Github which contains the library and is termed `kge`. The folder will be referenced as `/path/to/libKGE/kge` below.

#### 2. Download the pre-trained ComplEx model for CoDEx-M from the libKGE library

Download the file `codex-m-lp-complex.pt` from here, it contains the pre-trained model checkpoint. Place the file in this folder, i.e, the folder where this readme.html is located.

#### 3. Ensure you have all files.

Ensure the files `anyburl-test` and `anyburl-valid`, which contain the original AnyBURL rankings, are located in this folder, i.e, the folder where the readme.html is located.

You also need the location to the Codex-m dataset folder which is located in /path/to/libKGE/kge/data/codex-m

#### 4. Run the rescorex.py script, twice.

Ensure that you are within the python environment under which libKGE is installed. Navigate into this folder (alternatively, simply ensure that all the relativ/absolute paths to the files are correct) and run
`python rescorex.py --dataset_folder /path/to/libKGE/kge/data/codex-m --checkpoint codex-m-lp-complex.pt --ab_ranking anyburl-test --model_name complex`
subsequently, run
`python rescorex.py --dataset_folder /path/to/libKGE/kge/data/codex-m --checkpoint codex-m-lp-complex.pt --ab_ranking anyburl-valid --model_name complex`
When you are using Windows, additionally use the flag `-Xutf8`. After running the two commands, the two files `anyburl-test-complex` and `anyburl-valid-complex` have been created in the same folder. For help on the parameters of the script run `python rescorex.py --help`.

When running these scripts some warnings are generated. These warnings can be ignored. They are based on the fact that the checkpoints available at the libKGE page have been generated with an older libKGE version.

#### 5. Aggregate libKGE and AnyBURL ranking

The code for aggregating the ranking files is written in Java and uses AnyBURL as dependency. This means that you have to download the latest AnyBURL version, which is available as AnyBURL-JUNO.jar. The code for learning and performing the aggregation is available in the jar file JoinRE.jar located within this folder, i.e, the folder where this readme.html is located. Both jar files are compiled against the java 1.8 profile, which means that they should be also executable with older java versions.

Run the aggregation code with the following command (on Windows you have to separate the two jar files in the command with a ; semicolon, on Linux with a : colon). You have to specify seven input parameters seperated by blanks: (1) path to training file, (2) path to validation file, (3) path to ranking file created by a rule learner for the validation set, (4) path to rule-based ranking file with scores from kge model for the validation set, (5) path to ranking file created by a rule learner for the test set, (6) path to rule-based ranking file with scores from kge model for the test set, (7) output path where the aggregated ranking file should be stored.

You can use this command line call and run it from this folder, i.e, the folder where the readme.html is located. You have to change the first two arguments, to refer to your libKGE installation. The other parameters should fit to the content of this directory, if you conducted all previous steps.

`java -cp JoinRE.jar;AnyBURL-JUNO.jar x.y.z.Rescorer /path/to/libKGE/kge/data/codex-m/train.txt /path/to/libKGE/kge/data/codex-m/valid.txt anyburl-valid anyburl-test anyburl-valid-complex anyburl-test-complex output`

As results, an additional ranking file named `output` is stored in this folder.

#### 6. Evaluate the results

You can use the evaluation method from AnyBURL to evaluate it. This requires to modify the first three paths (which refer to training, validation and test set of the used evaluation dataset) to /path/to/libKGE/kge in the file `config-eval.properties`, which is also available in this folder. Compute hist@k and MMR with this command line call.

`java -cp AnyBURL-JUNO.jar de.unima.ki.anyburl.Eval`

The output should look like this:

    ...
    0.2772   0.3838   0.4922   0.3494

The numbers have the followng meaning: hits@1, hits@3, hits@10, and MRR.

## Reproducing results for other models / datasets

For all KGE models except of the transformer implementation we use the pre-trained models provided by libKGE. The AnyBURL rankings can be generated as described on the AnyBURL webpage. You can use the AnyBURL default parameter setting. There are only two exceptions:

- Please add the additioanl parameter `TOP_K_OUTPUT = 100` to the `config-apply.configuration`. This generates the top 100 rankings (10 in the default setting).
- As recommended by the AnyBURL developer, add MAX_LENGTH_CYCLIC = 5 to the `config-learn.configuration` for the WN18RR dataset. The default is 3, which shold be used for the other datasets.

## Training KGE models

As mentioned in the main text, for all KGE models except of the transformer implementation we use the pre-trained models provided by the libKGE library which can be downloaded here. For the transformer implementation, we provide detailed config information in the folder `transformer-configs` within the supplementary materials. All the configs can be run seamlessly under the current libKGE master (June 2021).

For the transformer model, we obtained hyperparameters for the FB237 dataset from the libKGE developers (`transformer-configs/training-configs/transformer-fb237-config.yaml`). For the remaining datasets, we use the Ax hyperparameter search provided by libKGE where we centered the search space around the FB237 parameters. For each dataset, we use the same search space. On Codex-L we run 15 trials out of which 7 are quasi-random. For the remaining datasets we use 30 trials with 15 quasi-random trials. The detailed specifications can be found in `transformer-configs/search-configs`.

The final models for the transformer are obtained by using the configurations of the search trials achieving the highest MRR on the validation split. The resulting configurations can be found in `transformer-configs/training-configs`.