Knowledge Distillation based Robot-Object Manipulation Failure Anticipation

Tugce Temel, Arda Inceoglu, Sanem SARIEL

Keywords: Failure Anticipation, Transformers, Knowledge Distillation

Abstract:

An autonomous service robot should be capable of interacting with its environment safely. However, it can encounter challenges during task execution due to uncertainties like perception errors, inaccuracies in manipulation, or unexpected external events. While most current research emphasizes detecting and classifying robot failures, our study shifts its focus to anticipating these failures beforehand. The underlying idea is that by anticipating a potential failure early on, preventive actions can be implemented. To address this, we present a
novel anticipation framework based on knowledge distillation. This system utilizes video transformers and incorporates a sensor fusion network designed to handle RGB, depth, and optical flow data. We assess the effectiveness of our method using a real-world robot manipulation dataset called FAILURE. Experimental results indicate that our proposed framework achieves an F1 score of 82.12%, highlighting its ability to anticipate robot execution failures up to one second in advance.

Submission Number: 13