# Invariant Representations Pipeline (Cats vs Humans)

This repository implements a frozen-encoder analysis pipeline across three model families to study cross-domain (human vs cat-filtered) representation invariance. It supports CNNs from torchvision, supervised transformers (ViT/Swin), and self-supervised transformers (DINO/DINOv2/DINOv3). The pipeline covers feature extraction, metrics computation (CKA, RSA/Mantel, distribution shift, paired similarity, multiple testing), and visualization utilities.

## Data Layout
- `data/frames/` — original images, organized in subfolders
- `data/cat_frames/` — cat-vision images, mirroring `frames/` subfolders
- Images are paired by filename within matching subfolder names across the two domains.

## Supported Model Families

### CNNs (torchvision)
- ResNet: `resnet18/34/50/101/152`
- ConvNeXt: `convnext_tiny/small/base/large`
- MobileNet: `mobilenet_v2`, `mobilenet_v3_small/large`
- DenseNet: `densenet121/169/201`
- EfficientNet: `efficientnet_b0` … `efficientnet_b7`

Loaded via torchvision multi-weight API (`DEFAULT` weights) with the corresponding preprocessing. Encoders are frozen and features are extracted via `create_feature_extractor` from internal nodes.

### Supervised Transformers (ViT/Swin)
- ViT: `vit_b_16`, `vit_l_16`
- Swin: `swin_t`, `swin_s`, `swin_b`

Vectorization policy: ViT hidden states are reduced to vectors using CLS token (or mean over tokens if CLS absent); Swin stage outputs are global-average pooled over spatial dimensions.

### Self-supervised Transformers (DINO / DINOv2 / DINOv3)
- DINO: `dino_vits16`, `dino_vitb16`
- DINOv2: `dinov2_small`, `dinov2_base`, `dinov2_large`
- DINOv3 (pretrain checkpoints): `dinov3_vitb16_pretrain`, `dinov3_vit7b16_pretrain`

Implemented via Hugging Face `transformers` with a lightweight wrapper that exposes hidden states as a dict of layers (`block{i}`), then vectorizes tokens as above. Encoders are frozen.

## Installation
1. Python 3.10+ recommended
2. Core requirements:
```
pip install -r requirements.txt
```
3. Optional (for DINO/DINOv2/DINOv3 models):
```
pip install transformers
```

## Command Line Interfaces

The pipeline is structured into three CLIs, one per family. All CLIs share a similar command set: `extract`, `analyze`, `viz2d`, and `vizmaps`.

### 1) CNNs
Entry point: `src/cli.py`
```
python -m src.cli <command> [options]
```
Commands and key options:
- extract: `--data_root`, `--out_dir`, `--models ...`, `--device`, `--golden_subset`, `--overwrite`, `--resume`, `--batch_size`
- analyze: `--embeddings_path`, `--out_dir`
- viz2d: `--embeddings_path`, `--method tsne|umap`, `--model ...`, `--layer ...`, `--out_dir`
- vizmaps: `--data_root`, `--model`, `--layer`, `--device`, `--channels`, `--max_pairs`, `--out_dir`

### 2) Supervised Transformers (ViT/Swin)
Entry point: `src/cli_transformers.py`
```
python -m src.cli_transformers <command> [options]
```
Commands and key options:
- extract: `--data_root`, `--out_dir`, `--models vit_b_16 vit_l_16 swin_t swin_s swin_b`, `--device`, `--golden_subset`, `--overwrite`, `--resume`, `--batch_size`
- analyze: `--embeddings_path`, `--out_dir`
- viz2d: `--embeddings_path`, `--method tsne|umap`, `--model ...`, `--layer ...`, `--out_dir`
- vizmaps: `--data_root`, `--model`, `--layer`, `--device`, `--channels`, `--max_pairs`, `--out_dir`

### 3) Self-supervised Transformers (DINO / DINOv2 / DINOv3)
Entry point: `src/cli_dino.py`
```
python -m src.cli_dino <command> [options]
```
Commands and key options:
- extract: `--data_root`, `--out_dir`, `--models dino_vits16 dino_vitb16 dinov2_small dinov2_base dinov2_large dinov3_vitb16_pretrain dinov3_vit7b16_pretrain`, `--device`, `--golden_subset`, `--overwrite`, `--resume`, `--batch_size`
- analyze: `--embeddings_path`, `--out_dir`
- viz2d: `--embeddings_path`, `--method tsne|umap`, `--model ...`, `--layer ...`, `--out_dir`
- vizmaps: `--data_root`, `--model`, `--layer`, `--device`, `--channels`, `--max_pairs`, `--out_dir`

## Typical Workflows

Example commands (adjust paths as needed):

### CNNs
```
# 1) Extract
python -m src.cli extract --data_root data --out_dir out_cnn --device cuda --overwrite
# 2) Analyze
python -m src.cli analyze --embeddings_path out_cnn/embeddings --out_dir out_cnn
# 3) 2D projection
python -m src.cli viz2d --embeddings_path out_cnn/embeddings --method tsne --out_dir out_cnn/viz
# 4) Feature maps
python -m src.cli vizmaps --data_root data --model resnet50 --layer layer3 --out_dir out_cnn/viz/maps
```

### Supervised Transformers
```
python -m src.cli_transformers extract --data_root data --out_dir out_transformers --models vit_b_16 swin_b --device cuda
python -m src.cli_transformers analyze --embeddings_path out_transformers/embeddings --out_dir out_transformers
python -m src.cli_transformers viz2d --embeddings_path out_transformers/embeddings --method umap --out_dir out_transformers/viz
python -m src.cli_transformers vizmaps --data_root data --model vit_b_16 --layer block10 --out_dir out_transformers/viz/maps
```

### Self-supervised Transformers (DINO)
```
python -m src.cli_dino extract --data_root data --out_dir out_dino --models dino_vitb16 dinov2_base --device cuda
python -m src.cli_dino analyze --embeddings_path out_dino/embeddings --out_dir out_dino
python -m src.cli_dino viz2d --embeddings_path out_dino/embeddings --method tsne --out_dir out_dino/viz
python -m src.cli_dino vizmaps --data_root data --model dino_vitb16 --layer block0 --out_dir out_dino/viz/maps
```

## Determinism & Resume
- `--golden_subset` yields a deterministic subset of 50 pairs for quick checks.
- Extraction is append-safe and will skip already written sample/model/layer rows unless `--overwrite` is passed. Use `--resume` to continue partially completed runs.

## Notes
- Device selection: pass `--device cpu` to force CPU.
- Feature vectorization:
  - CNN spatial features use global average pooling.
  - ViT uses CLS token if available, otherwise mean over tokens; Swin uses global average pooling over H×W.
