# Repeatability

- Transactions of Machine Learning Research
- Paper: Fully Automatic Neural Network Reduction for Formal Verification

Our results can be reproduced in one click 

## Installation
This folder contains the code as well as a docker file to run the AINNCS benchmarks in one click.

However, you need to provide a MATLAB Licence file `license.lic` to run the code:
- Create a MATLAB License file: 
	For the docker container to run MATLAB, one has to create a new license file for the container.
	Log in with your MATLAB account at https://www.mathworks.com/licensecenter/licenses/
	Click on your license, and then navigate to
	1. "Install and Activate"
    1. "View activated computers"
	1. "Activate a Computer"
	(...may differ depending on how your licensing is set up).
- Choose:
	- Release: `R2022a`
	- Operating System: `Linux`
	- Host ID: `0242AC11000a` (= Default MAC-Adress of Docker Container) | your host MAC Adress
	- Computer Login Name: `matlab`
	- Activation Label: `<any name>`
- When prompted if software is already installed, choose "Yes".
- Download the file and place it next to the docker file

If you are having trouble, don't hesitate contacting us.

## Run the code

You can run all results in one click in a docker container using the `submit.sh` script.

	./submit.sh
	
The results will be stored to `./results`.


Alternatively, open this directory in MATLAB, add everything the MATLAB path, and run:

	nn_neuron_merge_main()
	
Note that all required toolboxes have to be installed first (see <a href="https://cora.in.tum.de/manual">CORA Manual</a>):

- YALMIP Toolbox
- Symbolic Math Toolbox
- Optimization Toolbox
- Statistics Toolbox
- Multiple Precision Toolbox
- Deep learning Toolbox
- Deep Learning Toolbox Converter for ONNX Model Format

### Short vs. Long Evaluation

By default, a shorter evaluation is run, which takes about 1-2h.
This evaluation averages the verification result over `10` images.
To run the full evaluation, please open `./code/scripts/nn_neuron_merge_vnn_eran_eval_all.m` and set `MAX_N = 100`.
