#  On the Complexity of Verifying Quantized GNNs with Readout

This repository contains code, data, and notebooks for the article ''On the Complexity of Verifying Quantized GNNs with Readout''.
It includes experiments, result analysis, and visualizations aimed at understanding how quantization affects model performance and efficiency.

## Requirements

To run this code successfully, please ensure that:

- You have the correct version of **Python**:  
   Required: **Python 3.11.9**

- All required libraries are installed with compatible versions. You can do this by running:


`pip install -r requirements.txt`

## Project Files Overview --- Notebooks
### requirements.txt
Requirements to run a code with right versions of the libraries. Make sure you are using the correct Python version to run this code with PyTorch. You need **Python 3.11.9**, as not all PyTorch versions support all Python versions.

### experiments-with-tensors.ipynb
This notebook provides experiments with tensors using symmetric and asymmetric quantization schemes. It illustrates also the differences between INT8 and QINT8 formats.

### gnn-quantization-analysis-results.ipynb
This notebook presents a comprehensive analysis of quantization techniques applied to Aggregate Combined Graph Neural Networks with Global Readout (ACRGNNs).

Key components of this notebook include:
- Pre-Trained models: Evaluation of pre-trained GNN models.
- Post-Training Quantization (PTQ): Evaluation of dynamic quantization on pre-trained GNN models.
- Accuracy Metrics: Comparison of model performance before and after quantization.
- Visualization: Interactive plots illustrating quantization effects on accuracy and weight distributions.

The data for this experiments are presented on data_for_gnn_quantization_analysis_results.zip

### data_for_gnn_quantization_analysis_results.zip
This zip Archive stores 6 .csv files and one .xlsx:
- `Hints.xlsx` -- here we have 3 sheets:

    - Name of Datasets -- here we presented information that structured in the next way: Name of file | Information | Name in code | Affilation
    - Functions of plotting -- here we presented information that structured in the next way:Name of what we want to see plot| Function (with the input specification)
    - Functions for manipulating -- here we presented information that structured in the next way: Name of what we want to do |	Function (with the input specification)

- `non_qua_output_acrgnn.csv` -- Dataset that contains results about ACRGNN from Barcelò et al.
- `dymanic_quantized_results_size_time_pytorch_acrgnn.csv` -- Dataset that contains results about ACRGNN with Post-Training Dynamic quantization of weights with additional information about the time and size of model
- `model_sizes_original_dyn_qua_pytorch_all.csv` -- Dataset that contains results about ACRGNN with and without Post-Training Dynamic quantization of weights                                           
- `non_qua_output_tr_relu_acrgnn.csv` -- Dataset that contains results about ACRGNN with trReLU from Barcelò et al.
- `model_sizes_original_dyn_qua_pytorch_acrgnn_trrelu.csv` -- Dataset that contains results about ACRGNN with trReLU with and without Post-Training Dynamic quantization of weights
- `dymanic_quantized_results_size_time_pytorch_acrgnn_tr_relu.csv` -- Dataset that contains results about ACRGNN with trReLU  with Post-Training Dynamic quantization of weights with additional information about the time and size of model


### acrgnn-tables-plots.ipynb
This notebook presents an experimental analysis of how different activation functions — specifically ReLU and truncated ReLU — affect the performance of models trained on $\text{FOC}_2$ formulas.

We report accuracy metrics for both activation functions before and after applying dynamic Post-Training Quantization (PTQ).

The notebook includes plots titled: "Accuracy Distribution Across Activation Functions and Quantization Settings for p1, p2, p3 formulas", which visualize the comparative results.

The data for this experiments are presented on data_for_accuracy_acrgnn.zip

### data_for_accuracy_acrgnn.zip
This zip Archive stores 4 .csv files:
- `accuracy_relu.csv`
- `accuracy_relu_dptq.csv`
- `accuracy_trrelu.csv`
- `accuracy_trrelu_dptq.csv`

Inside all these files we presented the accuracy according to the number of Layers (in this case from 1 to 10) and accuracy: for Train, Test1, Test2 datasets for the p1,2,p3 formulas.

### kaggle.zip
This .zip contains both `data_for_gnn_quantization_analysis_results.zip` and `data_for_accuracy_acrgnn.zip`.


## Project Files Overview --- Code.py
This work builds on the GNN-Logic framework introduced by Barceló et al. [1], and follows the theoretical foundations from their ICLR 2020 paper on the logical expressiveness of GNNs [2].

### dataset.zip
Here is stored the datasets for the FOC_2 folmulas. PLease unzip them on foder `Supplement_materials\Code\src\data\datasets`

### explore_synthetich_data.py
This script collects the data used to generate the table titled "Dataset Statistics Summary" (Appendix: Experimental Data and Further Analyses). It provides minimum, maximum, and average statistics for synthetic datasets in terms of the number of nodes and edges.

### explore_ppi.py
This script supports the creation of two tables related to the PPI benchmark dataset:
- "Dataset Summary. PPI Benchmark" (Appendix: Experimental Data and Further Analyses) `DatasetSummaryppibenchmark.csv`
This table describes general statistics of the PPI dataset, including: Number of graphs, Node feature dimensions, Label dimensions, Average node degree. 

- "Dataset Statistics Summary. PPI Benchmark" (Appendix: Experimental Data and Further Analyses) `DatasetStatisticsSummaryppibenchmark.csv`
This table provides detailed statistical values, including minimum, maximum, and average values for the number of nodes and edges across graphs in the dataset.

### ppi_collect_traing_time.py
We collect the trainig time of the ACRGNN with activation function (ReLU) for the PPI Benchmark. Created a file `LayerTime.csv` that stores the running time of the PPI model during training process.


### run_ppi.py
In this file we refer to the code of the Barcelò et al. [1,2]. Authors described this file in the next way: The results will be printed to console and logged in `src/logging/results_ppi`. A single file will collect the last epoch for each GNN combination.
A file with no extension will be created with the mean of 10 runs for each configuration and the standard deviation.

We add the time measurements of the training time. The results will be printed in console as `Time taken for {l} layers: {running_time} seconds`. Table with all results of the running time can be formed after by file `ppi_collect_traing_time.py`.

We also provided the `README.md` file from the original code [1] inside `src` folder.

### run_ppi_dptq.py
This Python script takes the trained models generated by `run_ppi.py`, applies the dynamic Post-Traing Quantization (PTQ) [3,4], and collects the data for the Appendix.  
The models' results provided: Accuracy and Loss for the Train, Validation, and Test splits.  
Collected data presented by the following files: 
- `ppi_relu_results_for_appendix.log` -- Stores accuracy and loss for the original models from [1], across different numbers of layers. 
- `ppi_relu_quantized_results_for_appendix.log` -- Stores accuracy and loss for the dynamically quantized models, across different numbers of layers.
- `ppi_relu_results_size_for_appendix.log` -- Stores model size (in MB) for both original and quantized models, based on layer count.
- `ppi_relu_results_time_for_appendix.log` -- Stores inference time (in seconds) for both original and quantized models, based on layer count.  

All measurements follow the same procedure:
1. Load the pre-trained model.
2. Run train, validation, and test data through the model.
3. Record accuracy, loss, model size, and inference time.

### main.py
This script is based on and extends the original code by Barcelò et al. [1,2], with our own modifications to support dynamic quantization and batch evaluation of ACR-GNN models on synthetic datasets.

It is designed to generate models for datasets p1, p2, and p3, evaluate their performance, and optionally apply Post-Training Dynamic Quantization (PTQ) using PyTorch.

Key Features
-Loads pre-trained ACR-GNN models (FP32) from disk.
-Evaluates each model on Train, Test1, and Test2 splits.
-Optionally trains models from scratch if enabled.
-Applies PTQ to convert models to quantized versions (QINT8).
-Supports both ReLU and truncated ReLU activation functions.
-Uses torch.quantization.quantize_dynamic for quantization of nn.Linear layers.
-Evaluation results and quantization stats are printed to the console and saved to disk.
-Logs performance metrics and evaluation summaries.

### collect_data_before_dPTQ.py
This script processes `.log` files to extract evaluation metrics from the ACR-GNN models trained without quantization.
It collects the following information for each run: train_loss, test1_loss, test2_loss, train_micro, train_macro, test1_micro, test1_macro, test2_micro, test2_macro.
The extracted data is aggregated and saved to a CSV file `non_qua_output_(tr)relu_acrgnn.csv`

### collect_model_sizes_before_and_after_dPTQ.py
This scripts reads `.pth` files of the models with and without dynamic PTQ and collect the following information for each run: size,size_quantized,size_diff.
The extracted data is aggregated and saved to a CSV file `model_sizes_original_dyn_qua_pytorch_acrgnn_(tr)relu.csv`

### standardization _for_notebook.py
This utility script standardizes model paths in a `.csv` file by replacing local experimental directory prefixes with a unified structure.
It transforms entries in the model_name column to match the path format used in the Notebook section of this project, ensuring consistency between raw experiment outputs and paths referenced during analysis.
## Technical specification of the laptop
The experiments presented in this project were conducted on two different platforms:

- **Local Laptop**:  For `.py` a Samsung Galaxy Book4 laptop with an Intel Core i7-150U processor, 16 GB RAM, and 1 TB SSD storage. 
- **Kaggle Cloud Platform**.  For `.ipynb` notebooks executed on Kaggle: NVIDIA Tesla P100 GPU (16 GB RAM). Hosted runtime with pre-configured environment


## References
[1] Pablo Barceló, Egor V. Kostylev, Mikaël Monet, Jorge Pérez, Juan L. Reutter, and Juan Pablo Silva.  
**GNN-logic**, GitHub repository, 2021.  
Available at: [https://github.com/juanpablos/GNN-logic](https://github.com/juanpablos/GNN-logic)

[2] Pablo Barceló, Egor V. Kostylev, Mikaël Monet, Jorge Pérez, Juan L. Reutter, and Juan Pablo Silva.  
**The Logical Expressiveness of Graph Neural Networks**, 8th International Conference on Learning Representations (ICLR), 2020.  
Available at: [https://openreview.net/forum?id=r1lZ7AEKvB](https://openreview.net/forum?id=r1lZ7AEKvB)

[3] PyTorch Team.  
  **Dynamic Quantization Recipe**.  
  [https://docs.pytorch.org/tutorials/recipes/recipes/dynamic_quantization.html](https://docs.pytorch.org/tutorials/recipes/recipes/dynamic_quantization.html)

[4] PyTorch Documentation.  
  **Post-Training Dynamic Quantization**.  
  [https://pytorch.org/docs/stable/quantization.html#post-training-dynamic-quantization](https://pytorch.org/docs/stable/quantization.html#post-training-dynamic-quantization)
## License

This project is licensed under the [MIT License](LICENSE).