The deployment of robots in uncontrolled environments requires them to operate robustly un- der previously unseen scenarios, like irregular terrain and wind conditions. Unfortunately, while rigorous safety frameworks from robust optimal control theory scale poorly to high-dimensional nonlinear dynamics, control policies computed by more tractable “deep” methods lack guarantees and tend to exhibit little robustness to uncertain operating conditions. This work introduces a novel approach enabling scalable synthesis of robust safety-preserving controllers for robotic systems with general nonlinear dynamics subject to bounded modeling error, by combining game-theoretic safety analysis with adversarial reinforcement learning in simulation. Following a soft actor-critic scheme, a safety-seeking fallback policy is co-trained with an adversarial “disturbance” agent that aims to invoke the worst-case realization of model error and training-to-deployment discrepancy allowed by the designer’s uncertainty. While the learned control policy does not intrinsically guarantee safety, it is used to construct a real-time safety filter with robust safety guarantees based on forward reachability rollouts. This safety filter can be used in conjunction with a safety-agnostic control policy, precluding any task-driven actions that could result in loss of safety. We evaluate our learning-based safety approach in a 5D race car simulator, compare the learned safety policy to the numerically obtained optimal solution, and empirically validate the robust safety guarantee of our proposed safety filter against worst-case model discrepancy. See https://saferoboticslab.github.io/ISAACS/ for supplementary material.

ISAACS: Iterative Soft Adversarial Actor-Critic for Safety

Via operator theoretic methods, we formalize the concentration phenomenon for a given observable `$r$' of a discrete time Markov chain with `$\mu_{\pi}$' as invariant ergodic measure, possibly having support on an unbounded state space. The main contribution of this paper is circumventing tedious probabilistic methods with a study of a composition of the Markov transition operator $P$ followed by a multiplication operator defined by $e^{r}$. It turns out that even if the observable/ reward function  is unbounded, but for some for some $q>2$, $\|e^{r}\|_{q \rightarrow 2} \propto \exp\big(\mu_{\pi}(r) +\frac{2q}{q-2}\big) $ and $P$ is hyperbounded with norm control $\|P\|_{2 \rightarrow q }< e^{\frac{1}{2}[\frac{1}{2}-\frac{1}{q}]}$, sharp non-asymptotic concentration bounds follow. \emph{Transport-entropy} inequality ensures the aforementioned upper bound on multiplication operator for all $q>2$.  The role of \emph{reversibility} in concentration phenomenon is demystified. These results are particularly useful for the reinforcement learning and controls communities as they allow for concentration inequalities w.r.t standard unbounded obersvables/reward functions where exact knowledge of the system is not available, let alone the reversibility of stationary measure. 

Concentration Phenomenon for Random Dynamical Systems:  An Operator Theoretic Approach

We study the concentration phenomenon for discrete-time random dynamical systems with an un-
bounded state space. We develop a heuristic approach towards obtaining exponential concentration
inequalities for dynamical systems using an entirely functional analytic framework. We also show
that existence of exponential-type Lyapunov function, compared to the purely deterministic setting,
not only implies stability but also exponential concentration inequalities for sampling from the sta-
tionary distribution, via transport-entropy inequality (T-E). These results have significant impact
in reinforcement learning (RL) and controls, leading to exponential concentration inequalities even
for unbounded observables (i.e., rewards), while neither assuming reversibility nor exact knowledge
of the considered random dynamical system (assumptions at heart of concentration inequalities in
statistical mechanics and Markov diffusion processes).

Transportation-Inequalities, Lyapunov Stability and Sampling for Dynamical Systems on Continuous State Space

In this paper, we present a method for table tennis ball trajectory filtering and prediction. Our gray-box approach builds on a physical model. At the same time, we use data to learn parameters of the dynamics model, of an extended Kalman filter, and of a neural model that infers the ball's initial condition. We demonstrate superior prediction performance of our approach over two black-box approaches, which are not supplied with physical prior knowledge. We demonstrate that initializing the spin from parameters of the ball launcher using a neural network drastically improves long-time prediction performance over estimating the spin purely from measured ball positions. An accurate prediction of the ball trajectory is crucial for successful returns. We therefore evaluate the return performance with a pneumatic artificial muscular robot and achieve a return rate of 29/30 (97.7%).

L4DC 2023 Conference Submissions

ISAACS: Iterative Soft Adversarial Actor-Critic for Safety

Concentration Phenomenon for Random Dynamical Systems: An Operator Theoretic Approach

Transportation-Inequalities, Lyapunov Stability and Sampling for Dynamical Systems on Continuous State Space

Black-Box vs. Gray-Box: A Case Study on Learning Table Tennis Ball Trajectory Prediction with Spin and Impacts