FRanDI: Data-Free Neural Network Compression via Feature Regression and Deep Inversion

Konstantin Sobolev, Dmitry Ermilov, Nikolay Kozyrskiy, ANH-HUY PHAN

Keywords: Neural Network Compression, Zero-Shot, Knowledge Transfer

Abstract:

Contemporary post-training neural network compression methods make a model lighter and faster without a significant drop in performance. However, these methods heavily depend on the model's training data which might be unavailable in practical scenarios. In this work, we present *FRanDI*, a novel framework to enable post-training neural networks compression without data. Our method leverages the DeepInversion-based approach to generate synthetic data from the pre-trained model.
We propose a compressed network degradation teacher-student based recovery scheme called *Feature Regression*. In addition, we present a new proxy metric that correlates with the original model's target metric to evaluate model compression policies called *Output Discrepancy*. Our algorithm does not depend on the neural network's target task compared to other data-free methods. We evaluate our framework on three different neural network compression approaches: low-rank weight approximation, unstructured pruning, and quantization.

Submission Number: 82