In our attempt to solve Phase 2 of the Real Robot Challenge, we reinforcement learn a control policy, in simulation, conditioned on delivering a single chosen dice to a single chosen goal. To further simplify the problem faced by the policy, we provide it with privileged state information from the simulator. Upon transfer to the real robot, an image processing technique is employed to estimate this state information. Ultimately, Phase 2 proves significantly more difficult than Phase 1 and our method struggles to solve the task, both in simulation or indeed reality.

Real Robot Challenge Phase 2 - ThriftySnipe

In the second stage, we solved the task of using a real robot to manipulate the dice to a random target position. Each task is broken down into three stages: select the corresponding starting point and endpoint, move the robot finger to the contact points to the dice, and take the dice to desired positions. In order to locate the specific positions of the dice, we perform motion sampling on the X-axis and Y-axis, and fit the picture and position according to the least-squares method.

Real Robot Challenge Stage 2

We present an extension to our runner-up approach from the previous edition of the Real Robot Challenge. To solve the task of sequential goal reaching we focus on two aspects to achieving near-optimal trajectory: Grasp stability and Controller performance. In the simulated challenge, our method relied on a hand-designed Pinch grasp combined with Trajectory Interpolation for better stability during the motion for fast goal reaching. In Stage 1, we observe reverting to Triangular grasp to provide a more stable grasp when combined with Trajectory interpolation, possibly due to the sim2real gap. In future stages, we propose using sampling-based planning methods and multi-agent pathfinding algorithms for complex rearrangement tasks. The video demonstrations for our approach our available at https://youtu.be/dlOueoaRWrM.

Real Robot Challenge Stage 1: Cartesian Position Control with Triangular Grasp and Trajectory Interpolation

We extend heuristic grasping and motion planning used in RRC 2020 with impedance
control to reach the best performance of -33410. In future work, we plan to explore
data-driven grasp configuration selection mechanisms.

Real Robot Challenge 2021 - Stage 1 Report

Deep reinforcement learning has shown its advantages in real-time decision-making based on the state of the agent. 
In this stage, we solved the task of using a real robot to manipulate the cube to a given trajectory. The task is broken down into different procedures and we propose a hierarchical structure, the high-level deep reinforcement learning model selects appropriate contact positions and the low-level control module performs the position control under the corresponding trajectory. Our framework reduces the disadvantage of low sample efficiency of deep reinforcement learning and lacking adaptability of traditional robot control methods.
Our algorithm is trained in simulation and migrated to reality without fine-tuning. The experimental results show the effectiveness of our method both simulation and reality.

Real Robot Challenge Stage 1

In Phase 3, we use a mid-level policy with three separate low-level primitives to solve the cuboid manipulation task at the four difficulty levels. Each task is decomposed into three stages: moving the fingers to the cube to grasp it, rotating it on the table to minimize orientation error, and reposing it with three fingers. These grasping, lifting, and rotating primitives are sequenced with a state-machine that determines which primitive to execute given the current state, goal, and difficulty level. In our evaluation, our mid-level policy shows robust performance over multiple runs with randomized initial and goal positions.

Real Robot Challenge Phase 3 (sombertortoise)

In Phase 1, we presented an approach to rigid-body manipulation based on carefully-designed
motion primitives. In Phase 2, we demonstrated that these primitives combined with a more
robust state machine can reasonably complete all 4 tasks on a real robot. For Phase 3, to
perform manipulation with a cuboid, we continue with the same approach with a different
set of motion primitives. Our approach shows promising results on the position-based tasks
for cuboid manipulation. In general, our approach emphasizes the use of ML machinery only
when classical approaches fail.

Real Robot Challenge Phase 3: Manipulating Objects using High-Level Coordination of Motion Primitives

For  this  last  phase, we adapted our model-based cartesian impedance controller in three ways: An improved pre-grasping position controller, refined grasping points and obtaining new hyperparameters through Bayesian optimization on the real system. Combined, these three adaptations enabled us to tackle the significantly more challenging cuboid object, which required more delicate manipulation compared to the cube. Videos can be found at: https://sites.google.com/view/robotchallenge-modelbased

Model-based Cartesian Impedance Control (Phase 3)

Our approach combines motion planning with several scripted motion primitives to manipulate the cuboid. In the previous two phases we focused on learning corrections to our controller, but in this phase we did not have time to train and test the models. However, we find that our controller is able to complete the four tasks.
A video of our approach can be found at https://youtu.be/razH6-bvAPY

Real Robot Challenge Phase 3 (ardentstork)

In Phase 2, we solved the tasks of manipulating the cube to random goal positions with the real robot. Each task is decomposed into three stages: moving the fingers to the cube to grasp it, reposing the cube with three fingers, and flipping the cube with two fingers. These grasping, lifting, and flipping primitives are sequenced with a state-machine. As a baseline, we designed model-based controllers to execute each of these primitives. To improve performance of the controllers on the real robot, we trained a residual policy, which adds joint-torque corrections to torques output by the controller to mitigate discrepancies between the simulated and real robot.

Hybrid Policy Learning for Dexterous Manipulation

Learning to control a robot by directly applying model-free Reinforcement Learning (RL) is prone to fail due to extreme sample inefficiency. We propose to address this issue by employing several techniques to improve sample complexity. In simulation we employ reward shaping, multi-task learning, and apprenticeship learning. To transfer the learned policy to the real robot we use domain randomization techniques to improve the robustness of the learned policy. In subsequent phases we plan to use learned domain randomization to target performance on the real system rather than robustness.

Dexterous Manipulation by Multi-Task RL and Domain Randomization: Phase 2 Report

In Phase 1, we presented an approach to rigid-body manipulation based on carefully-designed motion primitives. We now demonstrate in Phase 2 that these primitives combined with a more robust state machine can reasonably complete all 4 tasks. For Phase 3, we propose using model-based and time-constrained (e.g. contextual bandit) RL to select and optimize primitives based on the environment. In general, our approach emphasizes the use of ML machinery only when classical approaches fail.

Video Attachment Playlist: https://www.youtube.com/playlist?list=PLEYg4qhK8iUaXVb1ij18pVwnzeowMCpRc

This playlist includes one video of a successful attempt for each difficulty level.

Real Robot Challenge Phase 2: Manipulating Objects using High-Level Coordination of Motion Primitives

For the real robot challenge, we combine a fast zero-order optimization method (iCEM) with model learning. Our iCEM method generates strong expert trajectories using a learned forward dynamics model, which is iteratively trained on newly collected data. We use parallel forward model ensembles and use their uncertainty as a cost penalty in the optimization. Through the tight coupling between all components, our algorithm runs in real-time, rendering it suitable for real robotic applications.

A Planning Approach with Learned Ensemble Models

For this phase, our model-based cartesian impedance was improved three ways: A refined, holistic formulation, task decomposition for complex manipulations and Bayesian optimization of the hyperparameters.
Combined, these three additions dramatically improved grasping and manipulation performance on the real system.
Videos: https://sites.google.com/view/robotchallenge-modelbased

Model-based Cartesian Impedance Control

Our approach is based on residual policy learning. We create a hand-designed control policy and learn residual corrections to this policy. The hand-designed policy consists of a position controller that follows a planned motion and a torque controller that acts to apply corrective forces to the cube. We try a series of grasps until we find one that is able to plan to the goal location. For Task 4, we additionally perform a set of scripted manipulations to align the orientation of the cube with the goal before executing our policy. Due to time constraints, our approach is only slightly changed from Phase 1 of the competition and we use the residual models trained for phase one without modification on Levels 2, 3, and 4. See the future work section for details about improving our training pipeline for the real system.

RCC 2020 Challenge Submissions