Marten: A Built-in Security DRL-Based Congestion Control Framework by Polishing the Expert

Zhiyuan Pan, Jianer Zhou, Xinyi Qiu, Weichao Li, Heng Pan, Wei Zhang

Abstract: Deep reinforcement learning (DRL) has been proved to be an effective method to improve the congestion control algorithms (CCAs). However, the lack of training data and training scale affect the effectiveness of DRL model. Combining rule-based CCAs (such as BBR) as a guide for DRL is an effective way to improve learning-based CCAs. By experiment measurement, we find that the rule-based CCAs limit the action exploration and even cause DRL’s excessive dependence to gain higher DRL’s reward gain. To overcome the constraints, we propose Marten, a framework which improves the effectiveness of rule-based CCAs for DRL. Marten uses entropy as the degree of exploration and uses it to expand the exploration of DRL. Furthermore, Marten introduces the shielding mechanism to avoid wrong DRL actions. We have implemented Marten in both simulation platform OpenAI Gym and deployment platform QUIC. The experimental results in production network demonstrate Marten can improve throughput by 0.36% and reduce latency by 14.89% on average compared with Eagle, and improve throughput by 2.79% and reduce latency by 11.73% on average compared with BBR.