Let's Be Realistic: AI-Recommender Use in a Complex Management Setting

Jana Gonnermann-Müller

Abstract:

Labels like "AI-powered" or "Human-Expert" activate mental models and shape user decisions. Yet, the transferability of these labels on
performance in complex, realistic tasks needs investigation. This study examines how recommender labeling and human factors
(mindset, expertise) impact performance in a complex business management scenario. We conducted an online experiment employing a management dashboard, where participants (N = 395) received recommendations labeled as either Artificial Intelligence (AI) or
Human-Expert-generated. Unlike previous research, labeling did not significantly influence task performance. Instead, graph literacy
and cognitive load were key predictors of performance. Participants with positive attitudes toward AI found recommendations helpful,
but their performance did not improve with their use. Expertise seems to be dominant in AI labeling in this context. These findings
highlight the interaction between expertise, mindset, and labeling, advocating for further research investigating in which contexts
labeling and human factors critically influence performance when using AI recommendations.