# 🧠 Bridging Brains and Concepts: Interpretable Visual Decoding from fMRI with Semantic Bottlenecks

This repository presents an **interpretable brain-decoding pipeline** that first converts fMRI signals into human-readable concepts and then reconstructs the visual stimulus.  
A **semantic bottleneck** makes every step voxel-level interpretable while preserving high reconstruction quality.

---

## 💡 Motivation  

Typical decoders map brain activity $y$ straight into a high-dimensional latent space $z$ (e.g. CLIP) and feed that to a diffusion model $g$ (a conditional latent diffusion model, e.g. VersatileDiffusion or StableDiffusion) to obtain a plausible reconstruction $\tilde{x}$ of the stimulus $x$:


$$
\tilde z \;=\; yW,
\qquad
\tilde x \;=\; g(\tilde z).
$$

Although the mapping from $y$ to $z$ is linear, the CLIP features $z$ are learned by deep networks and are not interpretable in the first place.  
We ask:

> **How can we make brain decoding semantically explicit?**

---

## 🧠 Method Overview  

### 1. Classical decoder (opaque)

Classically, neural data $y$ is mapped to an estimation $\tilde{z}$ of latent embeddings (in this case we deal with CLIP embeddings) via a linear mapping with weights $W$: 

$$
\tilde z \;=\; yW,
\quad
W \in \mathbb{R}^{\,n_{\text{vox}} \times d_{\text{CLIP}}}.
$$

### 2. Our decoder (interpretable)

We factor $W$ into two matrices:

$$
W \;=\; A\,B,
\quad
A \in \mathbb{R}^{\,n_{\text{vox}} \times a},\;
B \in \mathbb{R}^{\,a \times d_{\text{CLIP}}},
$$

with $a = 214$ **semantic dimensions**. Essentially we add a semantic bottleneck, first mapping with matrix $A$ the brain features to an interpretable space $\mathcal{L}$ of $214$ dimensions, and then learning the map $B$ from $\mathcal{L}$ to the CLIP space. 


$A$ (brain → concepts) Each column links voxels to a binary question
(“Is there a person?”, “Is it outdoors?” …).

$B$ (concepts → CLIP) Maps the 214-D concept vector into CLIP space.

Because every dimension of $\mathcal{L}$ answers a specific question, the study of matrix $A$ allows us to reveal voxel-level concept maps.


## 📁 Repository Structure

- `get_questions.ipynb`:  
  Extracts image captions for all training images. These were used by GPT-4 to generate 214 binary visual questions.

- `question_answering.ipynb`:
  Uses BLIP-2 to answer questions per image, forming interpretable embeddings. Each image is represented by a vector where each dimension is the binary answer to one question. The questions are divided in two files: `binary_questions.csv` and `detailed_questions.csv`. The output (interpretable embeddings for all the subjects) is available in the folder `int_embeds`.

- `interpretable_brain_decoding.ipynb`:  
  This is the main notebook and it contains the code used to:

  1 - Assesses the quality of the chosen embeddings as described in the article. This uses files present in the `clip_embeddings` folder, i.e. the CLIP-Vision embeddings of the NSD images.

  2 - Train the original BrainDiffuser for the selected $4$ subjects.

  3 - Train the intermediate model that maps interpretable embeddings to the CLIP space ("Int to CLIP" model)

  4 - Trains the two-step interpretable BrainDiffuser: fMRI → $\mathcal{L}$ → CLIP.

  Pretrained versions of the BrainDiffuser and Int to CLIP Diffuser are available, for all subjects, in the folder `models`.

- `decoding.py`: 
  This file contains the majority of the code used in the main notebook. Helper functions are present in `train_helpers.py`, `vae_helpers.py`, `vae.py`, `versatile_diffusion_dual_guided_embeddings.py` and `versatile_diffusion_dual_guided.py`. 

- `eval_metrics.ipynb`: 
  Notebook containing the code used to evaluate reconstruction performances as explained in the main article. 
- `ROI-Visualization.ipynb`:
  This notebook contains code to plot the $A$ matrix columns (i.e. voxel contributions) on the glass brains to reproduce the images present in the article.

---

## 🧪 Dataset

- **Natural Scenes Dataset (NSD)**  
  High-resolution 7T fMRI data with thousands of images from COCO  
  📎 https://naturalscenesdataset.org

---

## 📊 Results

- Only marginal drops in reconstruction performances, when compared to the original BrainDiffuser
- Semantic maps align well with known cortical regions (faces, places, bodies, words), enabling neuroscientific interpretation of concept-specific voxel patterns

---

## ⚙️ Requirements

See the file `requirements.txt`. 

## References 

Original BrainDiffusers code: https://github.com/ozcelikfu/brain-diffuser/tree/671f1403fe2a0515771c29d64fb839153cf12f5e 

VDVAE: https://github.com/openai/vdvae 

NSD Dataset: https://naturalscenesdataset.org/