# M8! When Would Memorization Metrics Mislead in Multi-Modal Models?

## Installation

``` bash
conda create -n mem python=3.10
conda activate mem
conda install pytorch==2.1.0 torchvision==0.16.0 pytorch-cuda=12.1 -c pytorch -c nvidia
pip install -r requirements.txt
pip install -e .
```

## Run

``` bash
python runners/run_subspace_prune.py --use_wandb
```

## Methods 

```
evaluation/
├── analysis.py                 ----> plot distribution and roc auc
├── evaluator.py                ----> MAIN LOGIC
├── gradio-magic.py             ----> funny gradio bug
├── inspector.py                ----> for manual labelling and visuals
├── metrics
│   ├── base.py
│   ├── be.py                   ----> Last token CA intensity + Localized NoiseDiffNorm
│   ├── ca.py                   ----> Cross Attention Entropy (across tokens)
│   ├── diversity.py            ----> SSCD Similarity + Tiled L2 Distance
│   ├── hessian.py              ----> Diff of Eigenvals of Hessians (2412.04140v2)
│   ├── __init__.py
│   └── noisediffnorm.py
└── utils.py

mvdream
├── attention_control.py        ----> be measurement + ca measurement + ca mitigation
├── attention_store.py          ----> deprecated, passive measurement only
├── attention_visualize.py      ----> cache attn maps to file to avoid memory explosion
├── configs
│   ├── sd-v1.yaml
│   └── sd-v2-base.yaml
├── editing
│   ├── nemo.py
│   ├── perturbation.py         ----> be mitigation + ip-* mitigation
│   ├── sail.py                 ----> paired with run_sail.py
│   ├── subspace_pruner.py      ----> paired with run_subspace_prune.py
│   └── uce.py                  ----> paired with run_uce_*.py
├── ldm
│   ├── models
│   │   │   ├── ddim.py         ----> controller, amg and intermediate artifact saving

runners/
├── run_amg.py                   ----> 2404.00922
├── run_input_perturb.py         ----> 2305.20086 'ip-rt', 'ip-rna', 'ip-cwr', 'ip-gni' + 2407.21720 'ip-wen'
├── run_nemo.py                  ----> 2406.02366
├── run_sail.py                  ----> 2412.04140
├── run_subspace_prune.py        ----> github.com/ruchikachavhan/editing-memorization
├── run_uce_fixed_concept.py     ----> 2308.14761
├── run_uce_multi_concept.py     ----> 2308.14761
└── run_xattn.py                 ----> 2410.21665 'be' + 2403.11052 'ca_entropy'

```

### References: Many thanks for your work!
https://github.com/somepago/DCR   
https://github.com/YuxinWenRick/diffusion_memorization   
https://github.com/renjie3/MemAttn   
https://github.com/chenchen-usyd/BE-PRSS   
https://github.com/ruchikachavhan/editing-memorization    
http://github.com/ml-research/localizing_memorization_in_diffusion_models/   
https://github.com/rohitgandikota/unified-concept-editing/   
https://github.com/chenchen-usyd/AMG   