Design Matters: Unveiling the Impact of AI Engineering Decisions on Alignment - A Case Study in School Dropout Prediction

Leticia Arco García, Johan Loeckx

Track: Type A (Regular Papers)

Keywords: AI engineering, Design decisions, Holistic alignment

Abstract: AI has achieved remarkable successes, yet 80% of proof-of-concept projects fail to reach production, encounter unintended side effects, or face ethical, legal, or societal backlash. Despite the extensive research from computational and sociological perspectives, there is a significant lack of rigorous studies that connect the two fields. In this paper, we claim that a holistic and interdisciplinary understanding of alignment is needed and that rigorous AI engineering plays a crucial role to improve quality/productivity. We present a survey of the current challenges that AI engineers face while designing AI systems in practice, drawing from both academic and grey literature. We show how innovation (people, tech, data, and process), business (feasibility, desirability, and viability), epistemic (theory, formalisation, and implementation), and lifecycle perspectives intersect, interact, and impact engineering decisions. However, there are still many open research questions: what a good AI engineering methodology should look like; what is missing; how to diagnose issues given the limited understanding of problems; how to translate contextual information into actionable design decisions; how to evaluate whether design choices truly improve outcomes; how to understand interdependencies and cascading effects across stages; and how to align quality, impact, and priorities when designing a solution. We illustrate our findings in a real application of predicting school dropout.

Serve As Reviewer: ~Leticia_Arco_García1

Submission Number: 69