Deep Actor-Critics with Tight Risk Certificates

Bahareh Tasdighi, Manuel Haussmann, Yi-Shan Wu, Andres R Masegosa, Melih Kandemir

Keywords: Reinforcement learning, PAC Bayes, Risk Certificates

TL;DR: We propose a practical method to certify the risk of deep actor-critic policies via Recursive PAC-Bayes bounds.

Abstract: After an period of research, deep actor-critic algorithms have reached a level where they influence our everyday lives. They serve as the driving force behind the continual improvement of large language models through user-collected feedback. However, their deployment in physical systems is not yet widely adopted, mainly because no validation scheme that quantifies their risk of malfunction. We demonstrate that it is possible to develop tight risk certificates for deep actor-critic algorithms that predict generalization performance from validation-time observations. Our key insight centers on the effectiveness of minimal evaluation data. Surprisingly, a small feasible of evaluation roll-outs collected from a pretrained policy suffices to produce accurate risk certificates when combined with a simple adaptation of PAC-Bayes theory. Specifically, we adopt a recently introduced recursive PAC-Bayes approach, which splits validation data into portions and recursively builds PAC-Bayes bounds on the excess loss of each portion's predictor, using the predictor from the previous portion as a data-informed prior. Our empirical results across multiple locomotion tasks and policy expertise levels.

Confirmation: I understand that authors of each paper submitted to EWRL may be asked to review 2-3 other submissions to EWRL.

Serve As Reviewer: ~Manuel_Haussmann1, ~Bahareh_Tasdighi1

Track: Regular Track: unpublished work

Submission Number: 20