The Role of Trust in AI-Augmented Business Process Management Systems

Giacomo Acitelli, Simone Agostinelli, Angelo Casciani, Andrea Marrella

Abstract: The significant impact of Artificial Intelligence (AI) on academia, industry, and government has led to a strong focus on the realization of trustworthy AI systems. Among them, there is an emerging class of process-aware information systems infused with AI, called AI-Augmented Business Process Management Systems (ABPMSs), which autonomously unfold and adapt the execution flow of business processes (BPs) through continuous conversation with their human principals, who oversee the system decision. While much research on trustworthy AI has been conducted on devising general-purpose trust recommendations, in this paper we take a first step toward exploring the role of trust to develop trustworthy ABPMSs. Specifically, we assess a relevant subset of trustworthy AI principles against the lifecycle stages of an ABPMS, thus providing a classification framework that identifies to which principles the ABPMS stages should conform. Then, we test the applicability of our framework on a real-world healthcare BP, and we evaluate its reliability through a user study involving 15 academics at the intersection of AI and BPM. The results show a promising consensus that our framework reasonably aligns trustworthy AI principles with the ABPMS stages.