NNCodec: An Open Source Software Implementation of the Neural Network Coding ISO/IEC Standard

Daniel Becking, Paul Haase, Heiner Kirchhoffer, Karsten Müller, Wojciech Samek, Detlev Marpe

Keywords: Neural Network Coding, Entropy Coding, Neural Network Compression, Local Scaling Adaptation, Open Source Software, MPEG Standards, Context-Adaptive Arithmetic Coding

TL;DR: First standard-conform implementation of the Neural Network Coding (NNC) ISO/IEC Standard, which deploys context-adaptive binary arithmetic coding to adjust the run-length code to neural network weight statistics on-the-fly.

Abstract:

This paper presents NNCodec, the first open source and standard-compliant implementation of the Neural Network Coding (NNC) standard (ISO/IEC 15938-17), and describes its software architecture and main coding tools. For this, the underlying distributions and information content of neural network weight parameters is analyzed and examined towards higher compression gains.
At the core of the coding engine is a context-adaptive arithmetic coder that adapts its binary probability models on-the-fly to weight statistics. We show that NNCodec achieves higher compression than Huffman code, that is commonly used for neural network compression, but also that the average codeword length of NNCodec is often below the Shannon entropy bound.
By introducing specifically trained local scaling parameters, NNCodec can compensate for quantization errors in the latent weight space to a certain degree, which we show experimentally for ResNets, EfficientNet, and a Vision Transformer network topology. The software and demo are available at https://github.com/fraunhoferhhi/nncodec.

Submission Number: 29