Quantization-aware Policy Distillation (QPD)

Thomas Avé, Kevin Mets, Tom De Schepper, Steven Latre

Keywords: DRL, Quantization, Distillation, Model Compression, Low-Power, Actor-Critic

TL;DR: We introduce a method based on quantization and policy distillation that can effectively compress a policy network down to 0.5% of its original size, without any loss in performance.

Abstract:

Recent advancements have made Deep Reinforcement Learning (DRL) exceedingly more powerful, but the produced models remain very computationally complex and therefore difficult to deploy on edge devices.
Compression methods such as quantization and distillation can be used to increase the applicability of DRL models on these low-power edge devices by decreasing the necessary precision and number of operations respectively. Training in low-precision is notoriously less stable however, which is amplified by the decrease in representational power when limiting the number of trainable parameters. We propose Quantization-aware Policy Distillation (QPD), which overcomes this instability by providing a smoother transition from high to low-precision network parameters. A new distillation loss specifically designed for the compression of actor-critic networks is also defined, resulting in a higher accuracy after compression. Our experiments show that these combined methods can effectively compress a network down to 0.5% of its original size, without any loss in performance.