Artifact for Paper: "Multi-Cost-Bounded Reachability Analysis of POMDPs"
by Alexander Bork, Joost-Pieter Katoen, Tim Quatmann, and Svenja Stein
==
# Introduction
This artifact contains the implementation for the paper "Multi-Cost-Bounded Reachability Analysis of POMDPs". Our implementation is integrated in the probabilistic model checker Storm (https://stormchecker.org).
Included are the source files of Storm with the extensions we made.
In addition to the tool, the artifact contains the model files we used for evaluating our implementation.
For validation of our original experiments, the original logfiles are also included.

# Installation
The installation process for our modified version of Storm is identical to building the regular version from source.

We refer to the installation guide (https://www.stormchecker.org/documentation/obtain-storm/build.html) for full information on the installation and provide a very brief guide here.

Make sure that all dependencies are available before installation. Our implementation requires no additional dependencies compared to the regular Storm version.

Required dependencies are:
- git
- cmake
- boost
- cln
- gmp
- ginac
- autoreconf
- glpk
- hwloc
- Z3

Afterwards, open a terminal and navigate to ``storm_cost_bounded_pomdp``.
Create a directory ``storm_cost_bounded_pomdp/build`` (using ``mkdir build``) and navigate to it (``cd build``).
To configure the installation, invoke 
```code
cmake ..
```
After this is done, compile the Storm binaries using
```code
make
```

# Running Storm

Storm is a command line tool, so to run it, 
Computing the cost-bounded reachability for a POMDP in Storm requires a model description in the PRISM language. For information on the language, we refer to the PRISM manual (https://www.prismmodelchecker.org/manual/ThePRISMLanguage/Introduction) and to the example models given in ``models``.

Note that to be consistent with the terminology used in Storm, we also refer to costs as rewards.

You can run ```./storm_cost_bounded_pomdp/build/bin/storm-pomdp --help``` or ```./storm_cost_bounded_pomdp/build/bin/storm-pomdp --help all``` to get more info on available command line switches for Storm. In the following, we outline the most important options for our work.

### Cost Unfolding
To use our implementation of cost unfoldings to check cost-bounded reachability queries, the general calls look as follows:
```
./storm_cost_bounded_pomdp/build/bin/storm-pomdp --prism path/to/model.prism --prop path/to/query.props _query-name_ --revised --unfold-reward-bound --belief-exploration {unfold --size-threshold _size_ | discretize --resolution _res_ --triangulationmode static}
```
where _query-name_ is the name of the query to check if multiple ones are present in the specified file.
This is the call used for configuration UNFOLD from the paper. The resulting cost-unfolding POMDP is then analysed using Storm's belief exploration engine, either to compute an under-approximation using the cut-off method (```--belief-exploration unfold```, configuration CUT from the paper) or an over-approximation using belief discretisation (```--belief-exploration discretize --triangulationmode static```, configuration DISCR from the paper). Both configurations have a hyperparameter each which is explained in Section "Configuring Belief Exploration".

### Cost-Aware/Level-Aware Unfolding
These calls are for configuration CA-UNFOLD from the paper. The POMDP is first transformed into a level-aware or cost-aware variant and the cost-bounded reachability query is then solved on the variant POMDP by cost unfolding.
The general call here is
```
./storm_cost_bounded_pomdp/build/bin/storm-pomdp --prism path/to/model.prism --prop path/to/query.props _query-name_ --revised --reward-aware {d_LIST} --unfold-reward-bound --belief-exploration {unfold --size-threshold _size_ | discretize --resolution _res_ --triangulationmode static}
```
where _query-name_ is the name of the query to check if multiple ones are present in the specified file.

``--reward-aware {d_LIST}`` specifies that the POMDP is first transformed into a cost-(level-)aware variant where ``{d_LIST}`` is an optional parameter which takes the form of a comma-separated list ``d_1,d_2, .., d_n`` whose length ``n`` is the number of cost dimensions referenced in the query. A value ``d_i`` corresponds to the width of the level, i.e., the $\mathbf{d}[i]$ value from the paper. A value 0 means that there is no additional observation in that dimension. If ``{d_LIST}`` is not given, the fully cost-aware variant of the POMDP is constructed.

### Sequential Approach on Cost-(Level-)Aware Variant
To run the sequential approach on a cost-(level-)aware variant of the input POMDP (configuration CA-BEL-SEQ), invoke
```
./storm_cost_bounded_pomdp/build/bin/storm-pomdp --prism path/to/model.prism --prop path/to/query.props _query-name_ --revised --reward-aware {d_LIST} --belief-exploration {unfold --size-threshold _size_ | discretize --resolution _res_ --triangulationmode static}
```
where _query-name_ is the name of the query to check if multiple ones are present in the specified file.
``--reward-aware {d_LIST}`` behaves as before.
First the cost-(level-)aware variant of the input POMDP is constructed. Then, an abstraction of its belief MDP is used to solve the cost-bounded reachability query using the sequential approach.

### Configuring Belief Exploration
To configure the belief exploration, we consider one hyperparameter each for CUT and DISCR.

CUT (```--belief-exploration unfold```):
- ```--size-threshold _size_``` sets the number of states ```_size_``` until which the belief MDP is explored. Afterwards, cut-offs are applied. If  ```_size_``` is 0, a heuristic threshold is computed and used as the threshold.

DISCR (```--belief-exploration discretize```):
- ``--resolution _res_ --triangulationmode static`` sets the resolution of the belief grid used for discretisation to integer ``_res_``.

Please refer to the logfiles for example calls on the models we used for the experimental evaluation.

# Logfiles
We provide all raw log files generated for our experimental evaluation in the folder 
``logs/raw``.
They document runs of our implementation for all settings with the hyper-parameters described in the appendix of the paper.
In addition to the raw log files (.log), there are also JSON files containing the relevant information extracted from the logs.

The files are named according to the scheme:
``storm.CONFIG C/D PARAM.EXPERIMENT_INSTANCE``
where ``CONFIG`` is one of the configurations 
- unf : UNFOLD,
- caunf : CA-UNFOLD,
- belseq : CA-BEL-SEQ
from the paper
and ``C/D`` indicates whether the run uses CUT ``(c)`` or DISCR ``(d)`` for approximation, followed by the considered hyper-parameter ``PARAM`` where for ``(c)`` the parameter is the exponent, i.e., we apply a size threshold of $2^\text{PARAM}$.
``EXPERIMENT`` takes value ``main``,``lvls``, or ``bnds`` and denotes the kind of experiment the logfile is referring to, where the latter two refer to the runs for the plots at the bottom of Figure 2 in the paper. All other results belong to ``main``.
Instances are named by the model identifier, followed by an (internally used) identifier of the bounded reachability query and the considered bound values. Note that the implementation uses non-strict inequalities for $>$.

### Interactive Table
As a more convenient way to view all results obtained during our experiments, we provide interactive tables. 

The tables are given in the HTML files
- ``logs/table/table.html`` (for the main experiments)
- ``logs/lvlstable/table.html ``(for the data used in the plot at the bottom left of Figure 2)
- ``logs/bndstable/table.html`` (for the data used in the plot at the bottom right of Figure 2)

and can be viewed in a web browser.
The columns of the tables are named similar to the naming scheme for the log files. In addition, we provide columns indicating the best result obtained within $10$, $100$, $1000$ and $1800$ seconds.
The latter coincides with the values considered in the main paper.
Columns can be hidden for a more clearly arranged view. 
A result cell contains the computed approximation values (also indicating if it is an under- or over-approximation) and the runtime (walltime) in seconds. Clicking on a result shows an overview as well as the raw log used to obtain the result.