Reinforcement learning has been successfully used to solve difficult tasks in complex unknown environments. However, these methods typically do not provide any safety guarantees, which prevents their use in safety-critical, real-world applications. In this paper, we attempt to bridge the gap between learning-based techniques that are scalable and highly autonomous but often unsafe and robust control techniques, which have a solid theoretical foundation that guarantees safety but often require extensive expert knowledge to identify the system and estimate disturbance sets. We combine a provably safe learning-based MPC scheme that allows for input-dependent uncertainties with techniques from model-based RL to solve tasks with only limited prior knowledge. We evaluate the resulting algorithm to solve a reinforcement learning task in a simulated cart-pole dynamical system with safety constraints.

Learning-based Model Predictive Control for Safe Reinforcement Learning

When deploying autonomous systems in uncertain environments, mismatch between a model of the system dynamics and the true dynamics is inevitable. For an autonomous agent to perform tasks in such settings, it must control a system subject to model uncertainty under external disturbances, and do so safely. Furthermore, this initial uncertainty might be too high to carry out the desired task safely, in which case autonomous agents must be able to learn, using data observed online to reduce the uncertainty about the system dynamics. Recent work in meta-learning has emerged as a promising, data-driven alternative for online learning, using an offline
training phase to imbue learning algorithms with prior knowledge needed to efficiently fit data observed online. Such approaches showed to outperform nonparametric kernel-based Gaussian processes in accuracy and adaptation capabilities.

In this work, we present a unified framework for safe learning and adaptive control of an uncertain nonlinear system which leverages the computational and data efficiency gains of a meta-learned dynamics model. This framework combines three key technical contributions: 
(1) Lipschitz normalization techniques to improve the uncertainty propagation properties of the meta-learned dynamics model; 
(2) a tractable optimization objective for the exploration phase; 
(3) formulations of exploration and exploitation tasks as chance-constrained optimal control problems, which are solved using a novel sequential convex programming algorithm.
In spite of an initially highly uncertain prior model causing the initial problem to be unfeasible, each phase is solved successively until the end region is safely reached.

Safe Learning and Control using Meta-Learning

We present a data driven approach to construct a library of feedback motion primitives for non-holonomic vehicles that guarantees bounded error in following arbitrarily long trajectories. This ensures that motion re-planning can be avoided as long as disturbances to the vehicle remain within a certain bound and also potentially when the obstacles are displaced within a certain bound. The library is constructed along local abstractions of the dynamics that enables addition of new motion primitives through abstraction refinement. We provide sufficient conditions for construction of such robust motion primitives for a large class of nonlinear dynamics, including commonly used models, such as the standard Reeds-Shepp model. The algorithm is applied for motion planning and control of a rover with slipping without its prior modelling. 

Data-driven construction of robust motion primitives for non-holonomic vehicles

Validation is a key challenge in the search for safe autonomy. Simulations are often either too simple to provide robust validation, or too complex to tractably compute. Therefore, approximate validation methods are needed to tractably find failures without unsafe simplifications. This paper presents the theory behind one such black-box approach: adaptive stress testing (AST). We also provide three examples of validation problems formulated to work with AST.

The Adaptive Stress Testing Formulation

We extend the learning from demonstration paradigm by providing a method for learning unknown constraints shared across tasks, using demonstrations of the tasks, their cost functions, and knowledge of the system dynamics and control constraints. Given safe demonstrations, our method uses hit-and-run sampling to obtain lower cost, and thus unsafe, trajectories. Both safe and unsafe trajectories are used to obtain a consistent representation of the unsafe set via solving a mixed integer program. Additionally, by leveraging a known parameterization of the constraint, we modify our method to learn parametric constraints in high dimensions. We show that our method can learn a six-dimensional pose constraint for a 7-DOF robot arm.

Learning Parametric Constraints in High Dimensions from Demonstrations

Partially observable Markov decision processes (POMDPs) are a natural model for scenarios where one has to deal with incomplete knowledge and random events.
Applications include, but are not limited to, robotics and motion planning.
However, many relevant properties of POMDPs are either undecidable or very expensive to compute in terms of both runtime and memory consumption.
In our work, we develop a game-based abstraction method that is able to deliver safe bounds and tight
  approximations for important sub-classes of such properties.
We discuss the theoretical implications and showcase the applicability of our results on a broad spectrum of benchmarks.


Strategy Synthesis in POMDPs via Game-Based Abstractions

The aim of this study is to introduce a formal framework for analysis and synthesis of driver assistance systems. It applies formal methods to the verification of a stochastic human driver model built using the cognitive architecture ACT-R, and then bootstraps safety in semi-autonomous vehicles through the design of provably correct Advanced Driver Assistance Systems. The main contributions include the integration of probabilistic ACT-R models in the formal analysis of semi-autonomous systems and an abstraction technique that enables a finite representation of a large dimensional, continuous system in the form of a Markov model. The effectiveness of the method is illustrated in several case studies under various conditions.