Transferring Human Daily Activity Skills to Surgical Robots via Deep Successor Features

TMLR Paper6110 Authors

Abstract: We propose a framework for surgical robot task learning that leverages large-scale human Activities of Daily Living (ADL) datasets to mitigate the scarcity of surgical training data. Surgical robot learning is uniquely constrained: datasets are limited, costly, and unsafe to collect at scale. In contrast, the robotics community has curated extensive ADL datasets capturing motor behaviors such as food preparation and tool use. Our key insight is that these datasets encode transferable visuomotor primitives—such as instrument manipulation and hand–eye coordination—that parallel the basic skills underlying surgical maneuvers. Inspired by how surgeons develop expertise by first mastering everyday skills before refining them in the operating room, we leverage ADL data to pretrain representations for surgical robot learning. To address task variability and embodiment differences, we design a modular deep successor feature architecture that learns predictive state representations from ADL tool-use and adapts them to surgical domains. Unlike prior approaches that depend solely on limited surgical data, our framework enables large-scale offline pretraining on abundant non-surgical datasets while supporting efficient reinforcement learning during deployment. We validate the framework on the da Vinci Research Kit (dVRK) in both simulation and real-world settings, showing that pretraining on ADLs accelerates adaptation with limited surgical data and improves sample efficiency compared to imitation learning and reinforcement learning baselines. While our current evaluation emphasizes a subset of fundamental surgical tasks, our results provide initial evidence that ADL pretraining offers a principled and scalable pathway toward data-efficient and safe autonomous surgical robot learning.

Submission Length: Long submission (more than 12 pages of main content)

Changes Since Last Submission: Revision in Fig.2

Assigned Action Editor: ~Li_Erran_Li1

Submission Number: 6110