# Activations analyzer

This experiment contains scripts to extract activations/weights from the model and 
calculate different statistics on them.

## How to use

### Data visualization

If you already have a csv file, then create a `csv` folder in `experiments/weights-extraction` and open `visualise.ipynb`. There you will be able to play with different plots.

### Weights extraction

The main script for extracting model's weights/activations is `extract.sh`. It will extract the weights and activations for both FP32 and HBFP formats. It will feed the given model with `$NUMBER_OF_SAMPLES` sentences from the `wikitext2` dataset and extract everything you need. Example:

```
MODEL="facebook/opt-125m" ./extract.sh
```

If you want to extract the activations for different models, launch `extract_multiple.sh`.

If you want to quantize only one particular layer, you can specify `$LAYER` and `$LAYER_TYPE` (e.g. `LAYER=5`, `LAYER_TYPE=self_attn.q_proj`).


### Statistics aggregation

After the weights and activations are extracted, we can calculate L2 error between FP32 and HBFP versions. For that, run `aggregate_statistics.sh` after you ran `extract.sh`. It will create a csv file `stats.csv` in the `csv` folder.

### Error propagation

To understand how error propagates through the network, we quantize only one layer and look at the L_2 error of each of the next layers in the network. To reproduce the expriment, run `extract_and_aggregate.sh`. After that you can visualize error propagation using `visualize.ipynb`.

### Technical note
This experiment doesn't use a GPU, it relies on a beefy CPU and a lot of RAM, because inference of one sentence is not that expensive.