Quantization-aware Policy Distillation (QPD)

Thomas Avé, Kevin Mets, Tom De Schepper, Steven Latre

Keywords: DRL, Quantization, Distillation, Model Compression, Low-Power, Actor-Critic

TL;DR: We introduce a method based on quantization and policy distillation that can effectively compress a network down to 0.5% of its original size, without any loss in performance.

Abstract:

Recent advancements have made Deep Reinforcement Learning (DRL) exceedingly more powerful, but the produced models remain very computationally complex and therefore difficult to deploy on edge devices.
Compression methods such as quantization and distillation can be used to increase the applicability of DRL models on these low-power edge devices by decreasing the necessary precision and number of operations respectively.
Training in low-precision is notoriously less stable however, which is amplified by the decrease in representational power when limiting the number of trainable parameters.
We propose Quantization-aware Policy Distillation (QPD), which overcomes this instability by providing a smoother transition from high to low-precision network parameters.
A new distillation loss specifically designed for the compression of actor-critic networks is also defined, resulting in a higher accuracy after compression.
Our experiments show that these combined methods can effectively compress a network down to 0.5% of its original size, without any loss in performance.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Reinforcement Learning (eg, decision and control, planning, hierarchical RL, robotics)