# Understanding Post-Training Structural Changes in Large Language Models

This project contains the key implementation code of the paper *Understanding Post-Training Structural Changes in Large Language Models*. 

You can execute the following code to install the required environment for Python:

```
conda create -n your_env_name python=3.12
conda activate your_env_name
pip install -r requirements.txt
```

`your_env_name` is your custom environment name. 

The project is structured as follows:
```
code
│   readme.md
│   requirements.txt
│   draw_attn_entropy.py
│   draw_cka.py
│   draw_svd_vector_sim.py
│   draw_SVSM.py
│   draw_orth.py
└───examples
    └───attn_entropy_results
    └───cka_results
    └───svd_results_14B
    └───orth_results
```

The code files provided here are the original source code used to generate the visualizations in the paper. Visualization results are stored in the `examples` folder, including: attention entropy analysis using GPQA dataset samples (`attn_entropy_results`), CKA heatmap visualizations (`cka_results`), and singular value decomposition examples from *Qwen2.5-14B* (`svd_results_14B`). The `orth_results` directory contains visualizations demonstrating orthogonality calculations using the normalized Frobenius norm.

`draw_SVSM.py` is responsible for drawing the *Singular Value Scaling Matrix (SVSM)* heatmap between two different models. The visualization generated by this code file is as follows:

<center class="half">
<img src="examples/svd_results_14B/base_reasoning_hot/3d_heatmap_q/svd__q_3d_heatmap.png" width=24%/>
<img src="examples/svd_results_14B/base_reasoning_hot/3d_heatmap_k/svd__k_3d_heatmap.png" width=24%/>
<img src="examples/svd_results_14B/base_reasoning_hot/3d_heatmap_v/svd__v_3d_heatmap.png" width=24%/>
<img src="examples/svd_results_14B/base_reasoning_hot/3d_heatmap_o/svd__o_3d_heatmap.png" width=24%/>
</center>

`draw_svd_vector_sim.py` is responsible for plotting the similarity, difference, orthogonality matrices between the singular vectors of two different models. The visualization generated by this code file is as follows:

<center class="half">
<img src="examples/svd_results_14B/base_reasoning_hot/25_dim/sim_0/svd_layer0_down_U.png" width=24%/>
<img src="examples/svd_results_14B/base_reasoning_hot/25_dim/sim_0/svd_layer0_down_V.png" width=24%/>
<img src="examples/svd_results_14B/base_reasoning_hot/25_dim/sim_0/svd_layer0_down_UV_Diff.png" width=24%/>
<img src="examples/svd_results_14B/base_reasoning_hot/25_dim/sim_0/svd_layer0_down_Orthogonality.png" width=24%/>
</center>

`draw_orth.py` is responsible for calculating the degree of orthogonality of the orthogonality matrix using the normalized frobenius norm. The visualization generated by this code file is as follows:

<center class="half">
<img src="examples/orth_results/orth_map_BaseVSBase_7B.png" width=32%/>
<img src="examples/orth_results/orth_map_BaseVSInstruct_7B.png" width=32%/>
<img src="examples/orth_results/orth_map_BaseVSReasoning_7B.png" width=32%/>
</center>

`draw_cka.py` is responsible for drawing CKA heatmaps after destroying and reconstructing the input or output subspace of models. The visualization generated by this code file is as follows:

<center class="half">
<img src="examples/cka_results/cka_heatmap_Instruct_14B_Q_0.png" width=24%/>
<img src="examples/cka_results/cka_heatmap_Instruct_14B_Q_1.png" width=24%/>
<img src="examples/cka_results/cka_heatmap_Reasoning_14B_Q_0.png" width=24%/>
<img src="examples/cka_results/cka_heatmap_Reasoning_14B_Q_1.png" width=24%/>
</center>

`draw_attn_entropy.py` is responsible for plotting the attention entropy of models under example inputs. The visualization generated by this code file is as follows:

!["attention entropy"](examples/attn_entropy_results/attention_entropy_Instruct_14B.png)

If you want to compare a series of phenomena between different models, you can modify the variables associated with the model name in the code file.
