# NTP4Verif

## Benchmarks

The total 5.3k extracted VCs are placed under
- `./data/why3/pearl` for the pearls of programs.
- `./data/why3/frama_c` for VCs from real C verification projects.

In detail:
- Contiki OS's Linked List Library: ./data/why3/frama_c/contiki_list/
- Contiki OS's Memory Allocator: ./data/why3/frama_c/contiki_memb/
- X.509 Parser: ./data/why3/frama_c/x509_parser
- Linux Kernel Scheduler's SWB: ./data/why3/frama_c/should_we_balance/
- Selected Cases from C++ STL: ./data/why3/frama_c/standard_algorithms/

### Test set

The cases selected into the test set are listed in the following files:
- `./test_set.isabelle.lst` for Isabelle
- `./test_set.roc.lst` for Rocq
- `./test_set.lean.lst` for Lean

### Source Why3 and C files

The source why3s files of the pearls are given under `./data/why3/pearl`, of suffix `.mlw`.
The source C code of the real C verification is given as tar-zst archives named `src.tar.zst` under each project's home directory, e.g., `./data/why3/frama_c/contiki_list/src.tar.zst`.

Additional to the above two folders, translations of Why3 system library are also required by these cases. These translations are placed under `./generation/<LANG>`.

In some languages (Lean, Rocq), a single VC is split into multiple statements due to the technique limitation. All statements placed in a single file are a single VC. This VC passes the test only when all the statements pass.

### Evaluation Results

The results are stored in ./evaluation/raw/
The raw responses are large and therefore stored in an anonymous Google drive:
    https://drive.google.com/file/d/1iuqaXDcQVa8CS7dvvqXJ874yCfYlFbts/view?usp=sharing

### How to proof check these benchmark cases

All the benchmark cases are configured with the package managers of the corresponding ITPs.

#### Isabelle
We use Isabelle's standard session management. You should run the following command to register the pakcages into the system.
```
# change the working directory to where this README.md is placed.
isabelle components -u.
```
And then you can open Isabelle/JEdit GUI to view and to proof check any extracted files.

### Rocq
We use `dune` to manage packages. Nothing needs to configure but you should use `dune build` to build a `.v` file, e.g.,
```
dune build ./data/why3/pearl/add_list_vcg/rocq/add_list_AddListImp_mainqtvc.vo
```

### Lean
We use the standard `lake` to manage our theories.
```
cd data/why3
lake build ./pearl/add_list_vcg/lean/add_list_AddListImp_mainqtvc.lean
```

## VC Extraction

The extraction pipeline is placed under NTP4Verif

The mapping & rewriting rules are given in
- ./lib/Isabelle/NTP4Verif/BuiltIn.yaml
- ./lib/Lean4Why3/BuiltIn.yaml
- ./lib/Rocq4Why3/BuiltIn.yaml

Some of the rules are hard-coded in ./NTP4Verif/lean.py, ./NTP4Verif/isabelle.py, ./NTP4Verif/rocq.py

### Setup the modified Why3 that dumps the AST

```
cd why3-for-NTP
opam init
opam switch create default ocaml
eval $(opam env --switch=default)
opam install menhir zarith
./configure --prefix $(realpath ../why3)
make -j
make install
```

### Extracting Why3 VCs from ACSL, by Frama-C

You must use Frama-C 18.0 (a rather old version).

The commands used to extract VCs in this benchmark are different per Frama-C project. The exact extraction command is given in the `src/extraction.sh`, e.g., `data/why3/frama_c/contiki_memb/src/extraction.sh` (after you unzip `data/why3/frama_c/contiki_memb/src.tar.zst`). Nonetheless, most of the commands are based on the following one:
```
frama-c -wp -wp-rte -wp-no-warn-memory-model -wp-gen -wp-out wp_out -wp-prover why3 <the-target-C-file>
```

### Translating Why3 VCs into Lean4/Rocq/Isabelle VCs

```
source envir.sh

# to extract pearl dataset for Lean4
./VCG.py lean data/why3/pearl --cache ./.lean.cache
# to extractpip install frama-c dataset for Lean4
./VCG.py lean data/why3/frama_c --cache ./.lean.cache -L ./lib/frama-c


# to extract pearl dataset for Rocq
./VCG.py rocq data/why3/pearl --cache ./.lean.cache
# to extract frama-c dataset for Rocq
./VCG.py rocq data/why3/frama_c --cache ./.lean.cache -L ./lib/frama-c

# to extract pearl dataset for Isabelle
ISA_REPL_ADDR=<IP-address-of-IsaREPL> ./VCG.py isabelle data/why3/pearl --cache ./.lean.cache
# to extract frama-c dataset for Isabelle
ISA_REPL_ADDR=<IP-address-of-IsaREPL> ./VCG.py isabelle data/why3/frama_c --cache ./.lean.cache -L ./lib/frama-c
```

### Run the Why3 ATP-pass Measurement

```
source envir.sh
./tools/why3_auto_3.py data/why3/pearl
# For frama-c cases, you must set the WHY3_LOAD=./lib/frama-c/ environment variable.
WHY3_LOAD=./lib/frama-c/ ./tools/why3_auto_3.py data/why3/frama_c/contiki_list
```


