Position: Evaluations of AI Moral Reasoning Still Miss Half of the Picture

Aidan Kierans, Ritam Dutt, Shiri Dori-Hacohen, Avijit Ghosh

Keywords: moral reasoning, construct validity, benchmarks, normative ethics, descriptive ethics, moral values, moral norms, AI alignment

TL;DR: AI moral reasoning benchmarks evaluate whether models share human values but not whether they can apply the moral norms those values require.

Abstract: Recent work on evaluating the moral competence of large language models (LLMs) has focused primarily on what we christen the moral value problem, i.e., do model outputs align with human moral values. In contrast, the moral norm problem, i.e. whether models can identify and correctly apply context-sensitive moral norms, remains underexplored. We posit that this imbalance stems partly due to the field’s reliance on descriptive ethics frameworks, such as Moral Foundations Theory and Kohlberg’s stages of moral development, which emphasize value representation over normative application. We review existing benchmarks and evaluation methods, and show that they cluster heavily around the value problem, while discussion regarding normative ethics remains underrepresented. We identify three crucial gaps : (i) the absence of high-quality ground-truth data for moral norms and their applications, (ii) insufficient evaluation of intermediate reasoning processes, and (iii) limited attention to the identification of morally relevant features in context. Subsequently, we propose a research agenda that includes the development of standardized formal representations for normative theories, the construction of expert-annotated datasets capturing norm application, and evaluation protocols that explicitly distinguish between values-level and norms-level competence. Our goal is to encourage more systematic study of normative reasoning in LLMs.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Type: Provocation

Archival Status: Non-archival

Submission Number: 82