Adaptive Virtual Fixtures for teleoperation often rely on visual inputs for online adaptation. State estimation from visual detections is never perfect, and thus affects the quality and robustness of adaptation. It is therefore important to be able to quantify how uncertain an estimation from vision is. This can, for example, inform on how to modulate a fixture's stiffness to decrease the physical force a human operator has to apply. Furthermore, the target of a manipulation operation might not be known from the beginning of the task, which creates the need for a principled way to add and remove fixtures when possible targets appear in the robot workspace. In this paper we propose an on-manifold Mixture of Experts (MoE) model that synthesizes visual-servoing fixtures while elegantly handling full pose detection uncertainties and 6D teleoperation goals in a unified framework. An arbitration function allocating the authority between multiple vision-based fixtures arises naturally from the MoE formulation. We show that this approach allows a teleoperator to insert multiple printed circuit boards (PCBs) with high precision without requiring the manual design of Virtual Fixtures to guide the robot motion. An exemplary video visualizing the probability distribution resulting from our model is available at: https://youtu.be/GKMQvbJ5OzA

Mixture of experts on Riemannian manifolds for visual-servoing fixtures

In the recent decade, there has been a rapid
growth in the field of assisted and autonomous driving with
companies like Tesla making strides in commercializing assisted
driving. Although commendable, automated driving systems
face a huge challenge in highly dynamic urban areas where
there are many interacting agents such as other vehicles and
pedestrians. Among those, pedestrian behavior is more dynamic
and therefore harder to predict. Furthermore, incorporating the
predicted behavior into decision-making is another challenge.
In this ongoing research, we propose Chance Constrained
Stochastic Shortest Path (CCSSP) incorporating probabilistic
multimodal trajectory prediction model to help solve those
orthogonal problems. The addition of probability to the mul-
timodal pedestrian trajectory predictions is crucial in making
informed and risk-bound decisions by an autonomous vehicle.

Chance-Constrained Stochastic Shortest Path For Autonomous Vehicle Navigation In Crowded Spaces

Particle filtering is a common technique for six degree of freedom (6D) pose estimation because of its ability to tractably represent belief over object pose. Due to the high-dimensional nature of 6D pose, this application of the particle filter is prone to particle deprivation and can lead to mode collapse of the underlying belief distribution in the importance sampling step.  When this occurs, recovering belief in the region surrounding the true state is challenging since it is no longer represented in the probability mass formed by the particles. Previous methods mitigate this problem by reinvigorating particles in the predicted belief, but determining the frequency of reinvigoration has relied on hand-tuning abstract heuristics. In this paper, we estimate the necessary reinvigoration rate at each time step by introducing a Counter-Hypothetical likelihood function, which is used alongside the standard likelihood. Inspired by the notions of plausibility and implausibility from evidential reasoning, the addition of our Counter-Hypothetical likelihood function assigns a level of doubt to each particle. The competing cumulative values of confidence and doubt across the particle set are used to estimate the level of failure within the filter, in order to determine the portion of particles to be reinvigorated. We demonstrate the effectiveness of our method on the rigid body object 6D pose tracking task.

Counter-Hypothetical Particle Filters for Single Object Pose Tracking

While Lidar-based SLAM systems are critical for autonomous vehicles, most existing methods are non-differentiable. This limits their use with deep neural networks to learn valuable representations. In this work, we propose Grad-Lidar-SLAM, a novel, fully differentiable SLAM framework for Lidar. We show its effectiveness by using it to solve point cloud completion task. We also show that when adapted for Lidar-based SLAM, existing differentiable baselines fail to perform well on real-world datasets. We address this problem by proposing a novel differentiable pose graph optimization framework. Our experiments on real-world datasets show that our proposed approaches outperform the baselines in static and dynamic environments.

Grad-Lidar-SLAM: Fully Differentiable Global SLAM for Lidar with Pose-Graph Optimization

Reliable scene understanding is indispensable for modern autonomous systems. Current learning-based methods typically try to maximize their performance based on segmentation metrics that only consider the quality of the segmentation. However, for the safe operation of a system in the real world it is crucial to consider the uncertainty in the prediction as well. In this work, we discuss the task of uncertainty-aware panoptic segmentation, which aims to predict per-pixel semantic and instance segmentations, together with per-pixel uncertainty estimates. We present two novel Evidential Panoptic Segmentation Networks, EvPSNet for solving this task with camera images, and EvLPSNet for LiDAR data. We provide several strong baselines combining state-of-the-art panoptic segmentation networks with sampling-free uncertainty estimation techniques for comparison. Extensive evaluations show that our approaches achieve the new state-of-the-art for the uncertainty-aware Panoptic Quality (uPQ) and the panoptic Expected Calibration Error (pECE). We make our code available at: https://github.com/kshitij3112

Uncertainty-aware Panoptic Segmentation for Camera and LiDAR Data

Many commodity sensors that measure the robot
and dynamic obstacle’s state have non-Gaussian noise char-
acteristics. Yet, many current approaches treat the underlying
uncertainty in motion and perception as Gaussian, primarily to
ensure computational tractability. On the other hand, existing
planners working with non-Gaussian uncertainty do not shed
light on leveraging distributional characteristics of motion and
perception noise, such as bias for efficient collision avoidance.

This paper fills this gap by interpreting reactive collision
avoidance as a distribution matching problem between the
collision constraint violations and Dirac Delta distribution. To
ensure fast reactivity in the planner, we embed each distri-
bution in Reproducing Kernel Hilbert Space and reformulate
the distribution matching as minimizing the Maximum Mean
Discrepancy (MMD) between the two distributions. We show
that evaluating the MMD for a given control input boils
down to just matrix-matrix products. We leverage this insight
to develop a simple control sampling approach for reactive
collision avoidance with dynamic and uncertain obstacles.

We advance the state-of-the-art in two respects. First, we
conduct an extensive empirical study to show that our planner
can infer distributional bias from sample-level information.
Consequently, it uses this insight to guide the robot to good
homotopy. We also highlight how a Gaussian approximation
of the underlying uncertainty can lose the bias estimate and
guide the robot to unfavorable states with a high collision
probability. Second, we show tangible comparative advantages
of the proposed distribution matching approach for collision
avoidance with previous non-parametric and Gaussian approx-
imated methods of reactive collision avoidance.

Leveraging Distributional Bias For Reactive Collision Avoidance under Uncertainty: A Kernel Embedding Approach

General robotic grippers are challenging to control because of their rich nonsmooth contact dynamics and the many sources of uncertainties due to the environment or sensor noise. 
In this work, we demonstrate how to compute 6-DoF grasp poses using simulation-based Bayesian inference through the full stochastic forward simulation of the robot in its environment while robustly accounting for many of the uncertainties in the system. A Riemannian manifold optimization procedure preserving the nonlinearity of the rotation space is used to compute the maximum a posteriori grasp pose. Simulation and physical benchmarks show the promising high success rate of the approach.

Simulation-based Bayesian inference for robotic grasping

The calibration of deep learning-based perception models plays a crucial role in their reliability. This work focuses on a class-wise evaluation of calibration measures to provide insights one challenge posed onto LiDAR-based semantic segmentation: classes with smaller instances and their subsequent underrepresentation in datasets. Those classes often include VRUs and are thus of particular interest for safety reasons.We evaluate the calibration of underrepresented classes in LiDAR-based semantic segmentation using a metric based on sparsification curves. Thereby we compare three semantic segmentation models of which one is specifically designed to better deal with smaller instances and underrepresented classes.

On the calibration of underrepresented classes in LiDAR-based semantic segmentation

We propose PK-ROKED, a novel probabilistic deep-learning algorithm to detect keypoints of a robotic manipulator in camera images and to robustly estimate the positioning inaccuracies w.r.t the camera frame.
Our algorithm uses monocular images as a primary input source and augments these with prior knowledge about the keypoint locations based on the robot’s forward kinematics.
As output, the network provides 2D image coordinates of the keypoints and an associated uncertainty measure, where the latter is obtained using Monte Carlo dropout.
In experiments on two different robotic systems, we show that our network provides superior detection results compared to the state-of-the-art.
We furthermore analyze the precision of different estimation approaches to obtain an uncertainty measure.


Robust Probabilistic Robot Arm Keypoint Detection Exploiting Kinematic Knowledge

Mobile inspection robots are typically tasked to find anomalies and report them, such that proper action can be taken. They operate in an open world, in which they encounter previously unseen situations in changing environments. We take the robot's goal, its context, prior knowledge and uncertainties into account to find anomalies that are relevant for the operation. Prior knowledge is expressed by logic formulas. E.g., a tool should not be left on the floor. These symbolic formulas describe anomalous objects in terms of predicates and variables (symbols) that can represent various concepts in the real world, their attributes and their relations. This knowledge can easily be adapted during the robot's operation to a new anomaly that is considered relevant. Reasoning is performed in a probabilistic, multi-hypothesis framework. A neuro-symbolic program evaluates the symbolic formulas against probabilistic, imperfect observations of the symbols. New anomalies require new symbols, which are measured in images by zero-shot language-vision models and their extensions for objects and segments. Starting from the symbolic formulas and predicates that describe the anomaly, our method infers what objects are involved that need to be detected, extracts the probabilistic information from the language-vision models and reasons about that via a neuro-symbolic program in order to find the anomaly of interest. Our contribution is the integration of the neuro-symbolic program and language-vision models. We show the effectiveness of our method to find anomalous situations in a robotic inspection setting.

Anomaly Detection in an Open World by a Neuro-symbolic Program on Zero-shot Symbols

Accurate localization is a critical requirement for
most robotic tasks. The main body of existing work is focused
on passive localization in which the motions of the robot are
assumed given, abstracting from their influence on sampling
informative observations. While recent work has shown the
benefits of learning motions to disambiguate the robot’s poses,
these methods are restricted to granular discrete actions and
directly depend on the size of the global map. We propose Active
Particle Filter Networks (APFN), an approach that only relies
on local information for both the likelihood evaluation as well as
the decision making. To do so, we couple differentiable particle
filters with a reinforcement learning agent that attends to the
most relevant parts of the map. The resulting approach inherits
the computational benefits of particle filters and can directly act
in continuous action spaces while remaining fully differentiable
and thereby end-to-end optimizable as well as agnostic to the
input modality. We demonstrate the benefits of our approach
with extensive experiments in photorealistic indoor environ-
ments built from real-world 3D scanned apartments. Videos and
code are available at http://apfn.cs.uni-freiburg.de.

IROS 2022 Workshop PRDL Submissions