# stochastic-order-eval

Repository for stochastic order evaluation


This repository provides code for relative almost stochastic dominance and absolute almost stochastic dominance testing and mean risk models computation. The repository is organized as follows:
- Stochastic Order Evaluation Library (soe)
    - Relative and Absolute Testing
    - Mean Risk Models
- Jupyter Notebooks 
    - Stochastic Order Evaluation on Mix Instruct DataSet 
    - Stochastic Order Evaluation on Toxicity DataSet 

Environment Set Up:
```conda create --name <env> --file requirements.txt```

The jupyter notebooks contain a segment of rank aggregation which has special environment restrictions.
To run ```MixInstruct_RankAggregation.ipynb``` and ```Toxicity_Combined_RankAggregation.ipynb``` create a new 
environment with the following information and instructions:

https://cran.r-project.org/web/packages/RankAggreg/vignettes/RankAggreg.pdf

https://academic.oup.com/bioinformatics/article/23/13/1607/223480

```bash
conda create --name rankaggreg_env python=3.8 -y
conda activate rankaggreg_env

pip install jupyterlab
pip install numpy
pip install matplotlib 
conda install -c r rpy2
```



To use the soe as a library in python scripts refer the following examples:


## Example: mean-risk models

```python
import numpy as np
from soe.meanrisk import MRM

# Generate some data
scores = np.array([1.83, 0.29, 0.75, 0.62, 1.72, 1.10])

# Instantiate the mean-risk models and compute the mean-risk scores
mean_risk_models = MRM(scores)
scores, info = mean_risk_models.compute()

info
>>> {'mean': 1.0516666666666667,
>>>  'p_eval': 0.1,
>>>  'delta': 0.24916666666666668,
>>>  'sigma': 0.378037696533031,
>>>  'gamma_X': 0.3148246666666666,
>>>  'h_X_p': 0.7906666666666669,
>>>  'gini_X_p': 0.6950606060606062,
>>>  'ntvar_X_p': -0.26099999999999995}

scores
>>> {'mean-delta': 0.8025000000000001,
>>>  'mean-sigma': 0.6736289701336358,
>>>  'mean-gamma_X': 0.7368420000000002,
>>>  'mean-h_X_p': 0.2609999999999999,
>>>  'mean-gini_X_p': 0.3566060606060606,
>>>  'mean-ntvar_X_p': 1.3126666666666666}
```


## Example: relative and absolute stochastic order test


```python
import numpy as np
from soe.testing import StochasticOrderTesting

k = 15  # number of models
n_samples = 100  # number of scores per model

# Generate random scores with a given random mean
means = np.random.permutation(k)
gt_rank = np.argsort(means)[::-1]

# For every model we have its scores in a list
scores_list = [m + np.random.randn(n_samples) for m in means]

test = StochasticOrderTesting(scores_list, n_bootstrap=100)


# Relative stochastic order test
rank_rel_qs, rank_rel_iqs = test.compute_relative_test(alpha=0.05)

# Check rank obtained with quantiles
rank_rel_qs == gt_rank
>>> array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
>>>         True,  True,  True,  True,  True,  True]) 

# Check rank obtained with integrated quantiles
rank_rel_iqs == gt_rank
>>> array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
>>>         True,  True,  True,  True,  True,  True]) 


# Absolute stochastic order test
rank_abs_qs, rank_abs_iqs = test.compute_absolute_test(alpha=0.05, tau=0.25)

# Check rank obtained with quantiles
rank_abs_qs == gt_rank
>>> array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
>>>         True,  True,  True,  True,  True,  True]) 

# Check rank obtained with integrated quantiles
rank_abs_iqs == gt_rank

>>> array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
>>>         True,  True,  True,  True,  True,  True]) 
```
