#BRAIN: Boltzmann Reinforcement for Analog Ising Networks

This repository contains the implementation of a novel variational reinforcement learning (VRL) framework for optimization on analog photonic Ising machines, demonstrating superior performance compared to traditional Metropolis-Hastings algorithms under realistic noise conditions.


*Manuscript in preparation*

## 🔬 Overview

The exponential scaling of combinatorial optimization problems demands computational paradigms beyond traditional digital architectures. Our VRL framework exploits the O(1) energy computation of photonic Ising machines while adapting to their inherent noise characteristics, achieving superior ground state discovery compared to conventional Monte Carlo methods.

### Key Features:
- **Noise-resilient optimization**: Maintains performance under up to 10% measurement noise
- **Superior scaling**: Favorable power-law scaling
- **Universal applicability**: Works with any Hamiltonian following Boltzmann distribution
- **Hardware compatibility**: Designed for analog Ising machines

## 📁 Repository Structure

```
├── RL/                                    # Variational Reinforcement Learning implementations
│   ├── all_to_all/
│   │   └── RL_BRAIN_a2a.py               # VRL for Curie-Weiss (all-to-all) model
│   └── nearest_neighbour/
│       └── RL_BRAIN_nn.py                # VRL for 2D nearest-neighbor Ising model
│
└── MH/                                    # Metropolis-Hastings implementations  
    ├── all_to_all/
    │   └── MH_BRAIN_a2a.py               # MH for Curie-Weiss (all-to-all) model
    └── nearest_neighbour/
        └── RL_MH_nn.py                   # MH for 2D nearest-neighbor Ising model
```

## 🚀 Quick Start

### Prerequisites

All dependencies are listed in the requirements.yml file. To create the environment, run:
```bash
conda env create -f requirements.yml
conda activate myenv  # Replace 'myenv' with the name specified in the .yml file
```

### Running VRL Algorithms

#### All-to-All (Curie-Weiss) Model
```python
# Navigate to RL/all_to_all/
python RL_BRAIN_a2a.py

# Key parameters (modify in script):
# - grid_shape: System size (default: 32x32 lattice)
# - noise_levels: Gaussian noise std (0.0 to 0.1)
# - beta_values: Inverse temperature β = 1/(T*J) (preferred range: 0.5 to 3.0)
# - n_steps_values: Training steps
# - steps_per_T_values: Number of variational steps at each training steps (default: 1)
# - n_samples_values: Number of samples to sample from Bernoulli distribution and compute gradient over (4000, for 32x32)
```

#### Nearest-Neighbor Model
```python
# Navigate to RL/nearest_neighbour/
python RL_BRAIN_nn.py

# Key parameters:
# - grid_shape: System size (default: 32x32 lattice)
# - noise_levels: Gaussian noise std (0.0 to 0.1)
# - beta_values: Inverse temperature β = 1/(T*J) (preferred range: 0.5 to 3.0)
# - n_steps_values: Training steps
# - steps_per_T_values: Number of variational steps at each training steps (default: 1)
# - n_samples_values: Number of samples to sample from Bernoulli distribution and compute gradient over (4000, for 32x32)
```

### Running Metropolis-Hastings Comparisons

#### All-to-All Model
```python
# Navigate to MH/all_to_all/
python MH_BRAIN_a2a.py

# Benchmark parameters:
# - lattice: .npy file is loaded corresponding to system size. List is given in MH/spin_configs folder. 
# - times: Number of Monte Carlo (MC) steps at each temperature (default: ~400000)
# - num_flips: Number of flips at each MC step (default: 1)
# - noise_levels: Gaussian noise std (0.0 to 0.1)
# - BJ_pairs: Define β pairs (start_β, end_β), β = 1/(T*J) (preferred range: 0.5 to 3.0)
```

#### Nearest-Neighbor Model
```python
# Navigate to MH/nearest_neighbour/
python RL_MH_nn.py


# Benchmark parameters:
# - lattice: .npy file is loaded corresponding to system size. List is given in MH/spin_configs folder. 
# - times: Number of Monte Carlo (MC) steps at each temperature (default: ~400000)
# - num_flips: Number of flips at each MC step (default: 1)
# - noise_levels: Gaussian noise std (0.0 to 0.1)
# - BJ_pairs: Define β pairs (start_β, end_β), β = 1/(T*J) (preferred range: 0.5 to 3.0)
```

## 🔧 Algorithm Details

### Variational Reinforcement Learning (VRL)

Our VRL framework combines:
- **Variational mean-field approximation**: Treats spin configurations as samples from learnable Bernoulli distributions
- **REINFORCE gradient estimation**: Policy gradient optimization with baseline subtraction


### Metropolis-Hastings (MH) Baseline

Standard implementation with:
- **Single-spin flip dynamics**: Sequential spin updates with Metropolis acceptance criterion
- **Detailed balance**: Ensures convergence to Boltzmann distribution (in noiseless case)


## 📊 Key Results

### Performance Comparison
- **Noiseless**: VRL requires ~7× more energy evaluations but provides superior exploration
- **3% Noise**: VRL achieves M=0.97 vs MH's M=0.5 at low temperature
- **Scaling**: VRL maintains N^0.98 (noiseless) and N^1.15 (noisy) scaling up to 65,536 spins

### Noise Resilience
- **MH failure**: Catastrophic breakdown even at 1% noise due to violated detailed balance
- **VRL robustness**: Maintains phase transition accuracy up to 10% noise
- **Critical temperature**: VRL correctly adapts to noise-induced disorder

## 🔬 Scientific Impact

### Computer Science Applications
- **Combinatorial Optimization**: MAX-CUT, graph partitioning, portfolio optimization
- **Machine Learning**: Neural architecture search, hyperparameter optimization
- **Quantum Computing**: Variational quantum eigensolvers, QAOA optimization

### Physics Applications  
- **Statistical Mechanics**: Phase transition analysis, rare event sampling
- **Condensed Matter**: Spin glass dynamics, frustrated magnetism
- **Materials Science**: Defect structure prediction, phase diagram mapping