Surgical instrument handling and retrieval in the operating room with a multimodal robotic assistant

Mithun George Jacob, Yu-Ting Li, Juan P. Wachs

Abstract: A robotic scrub nurse (RSN) designed for safe human-robot collaboration in the operating room (OR) is presented. The RSN assists the surgical staff in the OR by delivering instruments to the surgeon and operates through a multimodal interface allowing instruments to be requested through verbal commands or touchless gestures. A machine vision algorithm was designed to recognize the hand gestures performed by the user. To ensure safe human-robot collaboration, tool-tip trajectories are planned and executed to avoid collisions with the user. Experiments were conducted to test the system when speech and gesture modalities were used to interact with the robot, separately and together. The average system times were compared while performing a mock surgical task for each modality of interaction. The effects of modality training on task completion time were also studied. It was found that training results in a significant drop of 12.92% in task completion time. Experimental results show that 95.96% of the gestures used to interact with the robot were recognized correctly, and collisions with the user were completely avoided when using a new active obstacle avoidance algorithm.