# VeriCluX

VeriCluX is a program designed to generate, analyze, and validate explanations for clusters of inputs in neural network models. It provides tools for both generating explanations (for clusters and individual samples), evaluating their quality and performance, and running large-scale experiments on both local machines and computing clusters.

## Features
- **Cluster and Individual Explanations:** Generate explanations for clusters of data points or individual samples.
- **Performance Analysis:** Measure computation time and efficiency of explanation methods.
- **Quality Analysis:** Evaluate coverage, feature importance error, and feature ranking correlation of explanations.
- **Support for Classic and Custom Datasets:** Easily switch between standard datasets and user-defined custom datasets.
- **Batch and Local Execution:** Run experiments locally or submit large jobs to a cluster (e.g., SLURM).
- **Extensible:** Modular codebase for adding new explanation methods or datasets.

## Directory Structure
```
VeriCluX/
  core/            # Core logic: datasets, models, explanations
  data/            # Log files and explanation data
  figures/         # Generated plots and figures
    delta_robust/  # Delta-robust explanation figures
    cluster/       # Cluster explanation figures
  models/          # Trained model files (ONNX)
  paper/           # LaTeX paper and templates
  scripts/
    tools/         # Python execution scripts
    figures/       # Figure generation scripts (Bash)
    batch/         # Batch execution scripts (SLURM, etc.)
  tests/           # Pytest-based test suite
  README.md        # This file
  requirements.txt # Python dependencies
```

## Getting Started

### 1. Install Python
VeriCluX requires **Python 3.10**. It is recommended to use [pyenv](https://github.com/pyenv/pyenv) to manage Python versions.

```
pyenv install 3.10.12
pyenv virtualenv 3.10.12 vericlux-env
pyenv local vericlux-env
```

### 2. Install Dependencies
```
pip install -r requirements.txt
```

### 3. Running Scripts

#### Tools (Python Scripts)
- **Generate explanations:**
  ```sh
  python -m scripts.tools.generate_explanations --help
  ```
- **Train a model:**
  ```sh
  python -m scripts.tools.train --help
  ```
- **Classify a sample:**
  ```sh
  python -m scripts.tools.classify --help
  ```

#### Batch Scripts
- Submit a batch job (e.g., SLURM):
  ```sh
  sbatch scripts/batch/delta_robust/classic/generate_robust_explanations.sh
  sbatch scripts/batch/cluster/scalability/generate_cluster_explanations_by_num_features.sh
  ```

#### Figure Generation Scripts
- **Generate all figures:**
  ```sh
  bash scripts/figures/generate_all_figures.sh
  ```

### 4. Running Tests

VeriCluX uses `pytest` for testing. The test suite is organized into subdirectories by component type:

- `tests/analyzers/` — Tests for explanation analyzers
- `tests/factories/` — Tests for explanation factories
- `tests/validators/` — Tests for explanation validators

Each test file is named after the module/class it tests for clarity and maintainability. For more details, see `tests/README.md`.

To run all tests:
```
pytest tests -p no:warnings
```

You can also run specific categories or subdirectories, e.g.:
```
pytest tests/factories
pytest -m unit
```

## Replicating our Results

### Basic Workflow

To replicate the figures in the paper from our data, run

```
bash scripts/figures/generate_all_figures.sh
```

### Complete Workflow

To generate all the data from the ground up, proceed as follows.

1. **Setup the Environment**: Follow `Getting Started` to setup the environment.

2. **Train the Models**: On a computation cluster, run
    ```
    sbatch scripts/batch/train_custom.sh
    ```
    to train the models for the synthetic datasets, and run
    ```
    sbatch scripts/batch/train_classic.sh
    ```
    five times, manually changing the name of the internal `DATASET` parameter for each classic dataset.

3. **Clone the Log Data Structure**: Make the following folder structure
    ```
    data/logs
    data/logs/delta_robust
    data/logs/delta_robust/scalability
    data/logs/delta_robust/scalability/inflated-explanations
    data/logs/delta_robust/scalability/robust-explanations
    data/logs/delta_robust/classic
    data/logs/delta_robust/classic/inflated-explanations
    data/logs/delta_robust/classic/robust-explanations
    data/logs/cluster
    data/logs/cluster/scalability
    data/logs/cluster/scalability/cluster-explanations-by-cluster-size
    data/logs/cluster/scalability/cluster-explanations-by-num-features
    data/logs/cluster/scalability/simple-explanations
    data/logs/cluster/examples
    data/logs/cluster/examples/ordering-explanations
    data/logs/cluster/examples/simple-explanations
    data/logs/cluster/examples/cluster-explanations
    data/logs/cluster/classic
    data/logs/cluster/classic/simple-explanations
    data/logs/cluster/classic/cluster-explanations
    ```

4. **Generate the Explanations**:

    On a computation cluster, run the scripts in `scripts/batch/delta_robust` and `scripts/batch/cluster` to generate the explanations. The scalability scripts can be run as they are, but the scripts for classic datasets must be run five times, manually changing the name of the internal `DATASET` parameter for each classic dataset. Furthermore, make sure the parameters are as follows:

    | Dataset | NUM_LABELS | NUM_CLUSTERS |
    |---------|------------|---------------|
    | Diabetes | 2 | 5 |
    | Parkinsons | 2 | 5 |
    | Wine | 2 | 5 |
    | Glass | 6 | 2 |
    | Leaf | 10 | 1 |

5. **Process the Data**: Convert the raw log files into CSVs by running
    ```
    python scripts/tools/compact_data.py
    ```

6. **Generate the Figures**: Generate all the Figures in the paper by running
    ```
    bash scripts/figures/generate_all_figures.sh
    ```
    To run a particular figure, find the command in `scripts/figures/generate_robust_figures.sh` or `scripts/figures/generate_cluster_figures.sh` and run it.

## Contributing
Contributions are welcome! Please open an issue or pull request if you have suggestions, bug reports, or new features.
