Approximate Bisimulation Relation Restoration for Neural Networks Based On Knowledge Distillation

Jie Chen, Zihao Mo, Tao Wang, Weiming Xiang

Abstract: This paper employs knowledge distillation to optimize neural network compression processes via reducing the approximate bisimulation error between two neural networks. The paper calculates the approximate bisimulation error between two neural networks and derives the relationship between the approximate bisimulation error and the soft loss of knowledge distillation processes. Then, we propose a knowledge distillation optimization framework to further reduce the approximate bisimulation error between the original neural network and its compressed version. This method can significantly enhance the trustworthiness of the neural network compression methods as the approximate bisimulation error is reduced.