Safe exploration in reproducing kernel Hilbert spaces

Abdullah Tokmak, Kiran G. Krishnan, Thomas B. Schön, Dominik Baumann

Keywords: Bayesian optimization, safe reinforcement learning, reproducing kernel Hilbert spaces, robotics

TL;DR: We propose a safe BO algorithm that estimates the RKHS norm with statistical guarantees.

Abstract: Popular safe Bayesian optimization (BO) algorithms learn control policies for safety-critical systems in unknown environments. However, most algorithms make a smoothness assumption, which is encoded by a known bounded norm in a reproducing kernel Hilbert space (RKHS). The RKHS is a potentially infinite-dimensional space, and it remains unclear how to reliably obtain the RKHS norm of an unknown function. In this work, we propose a safe BO algorithm capable of estimating the RKHS norm from data. We provide statistical guarantees on the RKHS norm estimation, integrate the estimated RKHS norm into existing confidence intervals and show that we retain theoretical guarantees, and prove safety of the resulting safe BO algorithm. We apply our algorithm to safely optimize reinforcement learning policies on physics simulators and on a real inverted pendulum, demonstrating improved performance, safety, and scalability compared to the state-of-the-art.

Confirmation: I understand that authors of each paper submitted to EWRL may be asked to review 2-3 other submissions to EWRL.

Serve As Reviewer: ~Abdullah_Tokmak1

Track: Fast Track: published work

Publication Link: https://proceedings.mlr.press/v258/tokmak25a.html

Submission Number: 66