Future humanoid robots are expected to work in real environments such as extremely hazardous situations instead of humans. However, they still have difficulty in generating human-like behaviors, especially in terms of versatility and agility. 

To cope with this problem, we developed a Model Predictive Control (MPC) approach that takes full-body dynamics into account. MPC with full-body dynamics is a good candidate for generating a wide variety of agile movements because 
1. We do not have to design the low-level details of each movement. Such labor-intensive tasks are automated through numerical optimization.
2. Unlike using a highly reduced model of a humanoid robot, the optimization process under the constraints of full-body dynamics does not restrict the generable agile movements.

A well-known control approach is a hierarchical control architecture using an inverted pendulum and inverse dynamics control. However, an optimization process under such reduced model restricts the generable agile movements since the inverted pendulum model cannot take detailed limb movements into account. 

We evaluated our approach in skating tasks with simulated and real lower-body humanoids that have rollers on the feet. We are interested in generating wide variety of human-like agile motions. Then, we selected the skating movement generation as an illustrative task.  Our simulated robot generated various agile motions such as flipping down from a cliff, and our real lower-body humanoid also successfully generated a movement down a slope.

MPC for humanoid control

We published the journal paper Learning a Structured Neural Network Policy for a Hopping Task [1] roughly two years ago as a RAL journal paper with proceedings in IROS 2018. The paper is about learning a hopping motion on a single leg robot. The paper contributes a way to learn the dynamics of the system, how to optimize a hopping policy and two different ways to transfer the optimized policy to a neural network policy. The goal of one of the neural network policies, the feedback network policy, was to learn the feedback and feedforward gains. This allows to inspect the behavior of the policy by analyzing the outputs.
In the following, I outline a few lessons learned that are not mentioned in the original paper. In addition, I am listing a few things I would do differently from today’s standpoint.

Learning a Structured Neural Network Policy for a Hopping Task

In this work, we explore the use of Gaussian processes (GP) as function approximators for Reinforcement Learning (RL), and build estimates of the value function and Q-function using GPs. Such a representation allows us to learn Q-functions, and thereby policies, conditioned on uncertainty in the system dynamics, and can be useful in sample efficiently transferring policies learned in simulation to hardware. 
We use two approaches, GPTD and GPSARSA, to build approximate value functions and Q-functions respectively. While for simple, continuous problems, we found these to be effective at approximating the value function and the Q-function, for discontinuous landscapes GPSARSA deteriorates in performance, even on simple problems. As the problem complexity increases, for example, for an inverted pendulum, we find that both approaches are extremely sensitive to the GP hyperparameters, and do not scale well. We experiment with a sparse variant of the algorithm but find that GPSARSA still converges to poor solutions. Our experiments show that while GPTD and GPSARSA are nice theoretical formulations, they are not suitable for complex domains without extensive hyperparameter tuning.

Policy Iteration with Gaussian Process based Value Function Approximation

This paper provides a personal survey and retrospective of my work on
robot control and learning. It reveals an ideological agenda:
Learning, and much of AI, is all about making and using models to
optimize behavior. Robots should be rational agents. Reinforcement
learning is optimal control.  Learn models, and optimize policies to
maximize reward.  The view that intelligence is defined by making and
using models guides this approach.

This is a strong version of model-based learning, which emphasizes
model-based reasoning as a form of behavior, and takes into account
bounded rationality, bounded information gathering, and bounded
learning, in contrast to the weak version of model-based learning,
which only uses models to generate fake (imagined, dreamed) training
data for model-free learning.  Thinking is a physical behavior which
is controlled by the agent, just as movement and any other behavior is
controlled. Learning to control what to think about and when to think
is another form of robot learning.

That represents my career up to now. However, I have come to see that
model-free approaches to learning where policies or control laws are
directly manipulated to change behavior also play a useful role, and
that we should combine model-based and model-free approaches to
learning.

Everyday life is complicated. We (and robots) don't know what aspects
of the world are relevant to any particular problem. The
dimensionality of the state is potentially the dimensionality of the
universe. The number of possible actions is vast. The complexity
barrier for AI is like the speed of sound for airplanes, where we have
to punch through it as a field to go supersonic and develop useful
robots in our homes, daily lives, and on the road [The Right Stuff,
book 1979, movie 1983].

I advocate exploring life-long learning in the context of rich
knowledge, such as libraries of past observed, taught, actual, and
imagined behavior, rather than focusing on impoverished and isolated
learning where a single representation is adapted to do a single
behavior once from a single training set, and independent of any
larger context.  I advocate working with real robots in the real world
to address the complexity barrier.  Working with real robots is much
easier and more fun than trying to figure out how to get simulators to
capture the complexity that human learning effortlessly takes
advantage of.

What advice would I give a starting graduate student interested in robot learning?

Recent progress in AI technology has been breathtaking. However, many of the advances have played to the strengths of virtual environments: infinite training data is available, risk-free exploration is possible, acting is essentially free. In contrast, we require our robots to robustly operate in real-time, to learn from a limited amount of data, take mission- and sometimes safety-critical decisions and increasingly even display a knack for creative problem solving. To bridge this gap, here we offer an alternative view of recent advances in AI. In particular, we posit that, for the first time, roboticists can draw meaningful functional parallels between AI technology and components identified in the cognitive sciences as pivotal to robust operation in the real world: Dual Process Theory and metacognition. Revisiting recent work in robot learning, we establish the building blocks of a Dual Process Theory for robots and highlight potentially fruitful future research directions towards delivering robust, versatile and safe embodied AI.

Robots Thinking Fast and Slow: On Dual Process Theory and Metacognition in Embodied AI

This paper discusses the current development of research in touch sensing for robots, the major challenges, and the possible directions for future research. In the paper, I will briefly introduce the mechanism of humans' touch perception, which in years has provided lots of inspiration and guidance for roboticists to design tactile sensing frameworks. Apparently, the humans touch sensing capability is very powerful, mostly because it works as a system that integrates dynamic and static pressure sensing, kinesthetic perception, and the motion of limbs. Similarly, an intelligent touch sensing system for robots should also work as a \textit{system}, which contains multi-modal sensory input and integration with the robot's motion system. The questions are in both \textit{how} to build the system, and \textit{what} is the needed system. 

Touch Sensing for Robots: What is the Next?

Manipulation is having a moment. Over the last decade, academic and commercial players alike have renewed interest in robotic manipulation due to a variety of factors: increasing availability of robotic arms, standardization of grippers, and decreasing cost of compute power. One of the key challenges that remain is making manipulators robust and safe for interaction in unstructured environments. Tactile sensors, by augmenting visual perception with shape, texture, compliance, temperature, vibration, and/or force feedback, are a vital step towards making robotic manipulators useful outside the lab sandbox. As a reflection on tactile sensing for robotic manipulation, this paper has two goals: 1) to provide practical advice on choosing a tactile sensor design, and 2) to discuss prevalent sensing archetypes and trends through a historical lens. We conclude this paper with a discussion of future research questions and community goals given trends in tactile sensor design. 

From the Lab Notebook: Observations on Tactile Sensing for Robotic Manipulation

Benchmark is of vital importance to evaluate the effectiveness of an algorithm. In recent years, the computer vision community benefits from using standard datasets and benchmarks. This trend is becoming more popular in the robotics community as well.
However, if we would like to bring the same comparing approach to the robotics community, there are some challenges we have to face. Is there an alternative way to benchmark the performance of robots? We think using a simulated environment is the way out.
Humans study the physical world from the interaction with objects, and we believe robots should study in the same way. As building home assistant robots is our final goal, we started from building a dataset of articulated objects for robots to interact with.
Finally, with the help of the SAPIEN robotics kit that supports URDF and ROS, researchers have the possibility to train and test the robot in the simulator, and transfer it to the real world.
This paper also discusses the opportunities for improvement of the original paper, and proposes the new perspectives and future works since publishing.

Benchmarking Object-Centric Manipulation Using a Simulated Environment

In this retrospective,  a high-level principle to guide controller design for underactuated systems based on different treatment on null space will be disclosed, which is inspired by [4]. Firstly, we will show what is avoiding null space in underactuated systems. Secondly, we will introduce how this principle evolves. Thirdly, we will show the tricks that were used in the development. Finally, the limitations in [4] and corresponding extensions will be presented.

Utilizing Null Space in Overactuated Systems vs Avoiding Null Space in Underactuated Systems

Due to their small size and ability to follow nonlinear
paths, concentric tube continuum robots might offer new
opportunities for minimally invasive surgery. Currently, several
research groups in this relatively young field are working on
early-stage prototypes to develop technologies and methodologies
to realize this envisioned application. To date, a large variety of
different prototypes exists across the individual research groups.

In this paper, current habits of the research community are
investigated that cause such a proliferation of many different
prototypes. System thinking tools are applied to provide an indepth
analysis of the dynamics between publication pressure
and developing prototypes. According to a gathered data set
based on 139 publications in this research field, 53.2 % of the
publications include a robot-based evaluation. However, 61.1 %
of 36 prototypes are only used in one publication, indicating a
low prototype reuse rate. As a result, a lot of time is devoted
to create prototypes which hinders the development of a robotic
platform. Such a robotic platform can leverage the whole research
community by focusing efforts on the main research challenges.

CTCR Prototype Development: An Obstacle in the Research Community?

Human intent is often hard to model and predict. In fields such as robotics and biomechanics, one type of human motion that is important to model is the bipedal walking motion. As a type of legged locomotion, bipedal walking has unique advantages in daily-life environments such as stairs and muddy surfaces, compared to rolling locomotion that are used by many autonomous mobile robots. In this paper, recent developments in predicting bipedal gait dynamics and the corresponding human motion trajectory is presented. Such prediction usually require two main steps: data collection and data analysis. We inspect and compare existing solutions in each of the two steps, summarize the common approaches, and discuss the potential opportunities in the field going forward.

Recent Development in Human Motion and Gait Prediction

We address the problem of learning reusable state representations from a non-stationary stream of high-dimensional observations. This is important for areas that employ Reinforcement Learning (RL), which yields non-stationary data distributions during training. Unsupervised approaches can be trained on such data streams to produce  low-dimensional latent embeddings, which could be reused on domains with different dynamics and rewards. However, there is a need to adequately evaluate the quality of the resulting representations.
We propose an evaluation suite that measures alignment between the learned latent states and the true low-dimensional states. Using this suite, we benchmark several widely used unsupervised learning approaches. This uncovers the strengths and limitations of existing approaches that impose additional constraints/assumptions on the latent space.

RSS 2020 Workshop RobRetro Submissions