Auditing the bias of conversational AI systems in occupational recommendations: a novel approach to bias quantification via Holland's theory

Jacob Gelling, Aleksandra Urman, Aniko Hannak

Abstract: Recent research has utilized resumes and occupational profiles to identify and quantify discrimination within conversational AI systems. We extend this concept by proposing a novel approach to quantification which leverages Holland’s theory of occupational interests to provide a more holistic interpretation of discrimination in these systems. Building upon the seminal work of Bertrand and Mullainathan (Am Econ Rev 94(4):991–1013, 2004), which examined racial discrimination in hiring through a naturalistic experiment, we adapt their methodology to explore biases embedded in AI-driven job recommendations. By manipulating gender and race within user profiles, as well as investigating if these characteristics are inferred by name alone, we assess the extent to which conversational AI models (GPT-3.5, GPT-4, Llama 2, and Gemini) exhibit discriminatory patterns. Our findings indicate significant disparities in job recommendations based on demographic factors. These findings vary between AI models. We extend previous research in this domain which explores bias with singular metrics by transforming recommendations via Holland’s theory to a multidimensional set of characteristics allowing for a more thorough interpretation of bias. Our work underscores the importance of continuous auditing and refinement of AI systems to mitigate bias and promote fairness in automated decision-making processes.

External IDs:dblp:journals/jocss/GellingUH26