# SVGD Convergence Analysis

## Project Structure

The project is organized into directories, each representing a different experimental setup or target distribution:

```
.
├── BLR/
│   ├── blr_model
│   ├── blr_model.stan
│   ├── model.py
│   ├── plot.ipynb
│   ├── README.md
│   ├── run_mcmc.py
│   ├── running_experiments_decay.py
│   ├── SVGD.py
│   └── results/ (auto-generated when running experiments)
├── GM/
│   ├── model.py
│   ├── plot.ipynb
│   ├── README.md
│   ├── running_experiments_decay.py
│   ├── SVGD.py
│   └── results/ (auto-generated when running experiments)
├── MVN/
│   ├── model.py
│   ├── plot.ipynb
│   ├── README.md
│   ├── running_experiments_decay.py
│   ├── SVGD.py
│   └── results/ (auto-generated when running experiments)
├── .gitignore
├── LICENSE
├── README.md
└── .git/...
```

### Directory Contents:

* **`GM/`**: Contains code for experiments related to a Gaussian Mixture model.
* **`MVN/`**: Contains code for experiments related to a Multivariate Normal (Gaussian) model.
* **`BLR/`**: Contains code for experiments related to Bayesian Logistic Regression on the UCI Covertype dataset, including `blr_model` (compiled Stan model), `blr_model.stan` (Stan model definition), and a `results/` directory that will be automatically generated when running experiments.
* **`.gitignore`**: Git ignore rules for Python, R, and project-specific files.
* **`README.md`**: This file.

### Common Files:

Each model directory (`GM/`, `MVN/`, and `BLR/`) contains the following core files:

* **`SVGD.py`**:
    * Implements the core Stein Variational Gradient Descent (SVGD) algorithm.
    * Includes `svgd_kernel` for RBF kernel computations, and `update` method for performing SVGD iterations.
    * Calculates convergence metrics such as KL divergence and Kernelized Stein Discrepancy (KSD).
    * Supports different kernel modes (e.g., 'rbf', 'linear') and includes options for Adagrad optimization and learning rate decay.
    * Tracks eigenvalues of the kernel matrix, providing insights into the algorithm's behavior.

* **`model.py`**:
    * Defines the probabilistic model. For example, in `MVN/model.py`, it implements the `MVN` (Multivariate Normal) class with a `dlnprob` method to compute the gradient of the log-probability, which is essential for SVGD.
    * In `BLR/model.py`, it implements Bayesian Logistic Regression for multi-class classification on the UCI Covertype dataset.

* **`running_experiments_decay.py`**:
    * Scripts for executing the SVGD experiments.
    * It typically sets up initial conditions, runs the SVGD `update` method for a specified number of iterations, and saves the results (e.g., `kl_list`, `ksd_list`, `eig_list`) as pickle files (`.pkl`) in a `results/` subdirectory that is automatically created.
    * These scripts are configured to test various parameters, including the number of particles, step size, and different step-size decay factors (e.g., `decay_factor=0.1`, `beta=1`, `c=0.1`).

* **`run_mcmc.py`** (BLR only):
    * Script to run Markov Chain Monte Carlo (MCMC) simulations for the Bayesian Logistic Regression model.
    * Generates MCMC samples which can be used for comparison with SVGD results or for independent analysis of the posterior distribution.

* **`plot.ipynb`**:
    * Jupyter Notebooks designed for visualizing the experimental results.
    * They load the `.pkl` files generated by `running_experiments_decay.py`.
    * Plots include:
        * Cumulative mean of KSD vs. iterations.
        * KL divergence vs. iterations.
        * Final KSD and KL values as a function of the number of particles.
        * Analysis of step-size sensitivity and eigenvalues.
        * For BLR experiments: test/train accuracy comparisons and classification performance metrics.

## Experiments

The project includes experiments for:

* **Gaussian Mixture Models (`GM/`)**: SVGD convergence on a Gaussian mixture distribution.
* **Multivariate Normal Models (`MVN/`)**: SVGD convergence on a multivariate normal distribution.
* **Bayesian Logistic Regression (`BLR/`)**: SVGD convergence on a real-world multi-class classification problem using the UCI Covertype dataset.

Each experiment often involves:

* Varying the number of particles (e.g., 5, 10, 100, 1000 particles).
* Testing different step-size decay factors (e.g., 1.0, 0.67, 0.5, 0.0) to analyze their impact on convergence.

## How to Run (Inferred)

To replicate the experiments and generate plots:

1.  **Clone the repository:** (Assuming this code is from a repository branch)
    ```bash
    git clone <repository_url>
    cd svgd_convergence
    ```
2.  **Install dependencies:**
    ```bash
    pip install numpy scipy scikit-learn tqdm torch matplotlib seaborn
    ```
3.  **Run experiments:**
    Navigate to either the `GM/`, `MVN/`, or `BLR/` directory and run the `running_experiments_decay.py` script. This will automatically create a `results/` directory and generate the `.pkl` result files.
    ```bash
    cd GM/
    python running_experiments_decay.py
    ```
    (Repeat for `MVN/` and `BLR/` if desired)

4.  **Visualize results:**
    Open the `plot.ipynb` notebook in your preferred Jupyter environment (e.g., Jupyter Lab, Jupyter Notebook) and run the cells. The `results/` directory will be automatically created when you run the experiments, containing the `.pkl` files needed for visualization.
    ```bash
    jupyter notebook GM/plot.ipynb
    ```
    (Repeat for `MVN/plot.ipynb` and `BLR/plot.ipynb` if desired)

**Note:** The plotting notebooks expect a `results/` folder in the current working directory to load the data.

## BLR Experiment Details

The Bayesian Logistic Regression experiment (`BLR/`) is a new addition that tests SVGD on a real-world classification problem:

* **Dataset**: UCI Covertype dataset (forest cover type prediction)
* **Model**: Multi-class Bayesian Logistic Regression with Gaussian prior
* **Classes**: 7 different forest cover types
* **Features**: 54 features (topographic and spectral variables)
* **Evaluation**: Test/train accuracy and KSD convergence analysis

This experiment complements the synthetic experiments (GM and MVN) by providing insights into SVGD's performance on high-dimensional, real-world data.