DexNinja: Learning Robust Dexterous Cutting Policy with a Real-to-Sim-to-Real Data Engine

Haozhe Lou, Runyi Yang, Wanqi Zhong, Caiyun Liu, Yicheng Liu, Wenhao Liu, Wenlong Ma, Jiawei Xia, Xu Zheng, Danda Pani Paudel, Luc Van Gool, Hang Zhao, Yiming Li

Reviewer: ~Runyi_Yang1, ~Haozhe_Lou2

Keywords: Real2Sim2Real; Robot Learning; Grasping and Manipulation; Dexterous Hand

Abstract: Cutting a piece of food looks effortless in human hands: we nudge the blade, adjust the angle, modulate pressure as the skin yields and cuts. For a robot, however, every cut is a high-contact interaction with a deformable, elasto-plastic object whose behavior changes with pose, contact mode, and cutting stage. Collecting real-world data is also inherently expensive and wasteful: each trial typically contaminates, damages, or destroys the food, so obtaining scalable real world data for dexterous cutting are expensive and difficult. Simulation is an appealing alternative, but a large sim-to-real gap exists in both physics and perception. This gap is especially severe for topology-changing interactions like cutting, where small errors in contact, friction, or material and elastic transformations response quickly compound. We propose DexNinja, a differentiable real2sim2real framework that turns a handful of real demonstrations into a dense and realistic training distribution. DexNinja (i) reconstructs object instances from real trajectories, (ii) randomizes physically meaningful parameters (e.g., mass, geometry, friction) under category-level constraints, and (iii) augmented dexterous manipulation episodes using a custom differentiable simulator that couples robot dynamics, tactile contact, and deformable cutting. We evaluate DexNinja on a dexterous food-slicing task show that (1) the augmented data consistently improves sim-to-real transfer, and (2) the resulting policies generalize to out-of-distribution objects such as strawberries with varying shapes and sizes.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 20