Enabling Process Mining on Multimodal Robotic Data

Flavio Corradini, Sara Pettinari, Barbara Re, Lorenzo Rossi, Massimiliano Sampaolo

Abstract: Robotic systems are increasingly deployed across diverse domains, performing multiple operations while continuously interacting with the environment and making decisions. This results in large volumes of multimodal data from various sources, including sensor readings, video feeds, and communication data. In this domain, process mining holds the potential for extracting behavioral patterns, identifying anomalies, and evaluating performance metrics in robotic systems. However, its application remains largely unexplored due to the fine-grained, multimodal nature of robotic data. In this work, we investigate how robotic data should be processed to enable process mining applications. We explore activity recognition techniques to transition from robotic fine-grained data to high-level activities, applying Conditional Random Fields to sensor data and fine-tuning the Florence-2 model for video data. Furthermore, we outline key challenges and opportunities in preparing robotic data for process mining, laying the groundwork for future research on process mining-based analysis of robotic systems.