Fast Autoregressive Bit Sequence Modeling for Lossless Compression
Keywords: autoregressive probability estimation, computational effeciency, lossless compression, transformers, convolutional neural networks, GPUs
TL;DR: We propose basic components for autoregressive probability estimation of data sequences with DNNs to reduce the computational cost.
Abstract: Autoregressive probability estimation of data sequences is a fundamental task in deep neural networks and has been widely used in applications such as lossless data compression. Since it is a sequential iterative process due to causality, there is a problem that its process is slow. In this paper, we propose Scale Causal Blocks (SCBs), which are basic components of deep neural networks that aim to significantly reduce the computational and memory cost compared to conventional techniques. Evaluation results show that the proposed method is one order of magnitude faster than a conventional computationally optimized Transformer-based method while maintaining comparable accuracy.
Submission Number: 9
Loading