Do LLM Recommenders Obey Preference Axioms? Testing Logical Rationality in LLM-Based Recommendation

Alok Upadhyay

Track: long paper (up to 10 pages)

Keywords: LLM, RecSys, Logical Reasoning, Preferences, Ranking

TL;DR: Do LLM Recommenders Obey Preference Axioms? Testing Logical Rationality in LLM-Based Recommendation

Abstract: Large Language Models (LLMs) are increasingly used as zero-shot recommender systems, yet their outputs are evaluated almost exclusively on accuracy metrics. We ask a more fundamental question: are LLM-based recommendations logically rational? Drawing on classical social choice theory, we systematically test whether LLM recommenders satisfy four foundational preference axioms---transitivity, Independence of Irrelevant Alternatives (IIA), and Sen's contraction ($\alpha$) and expansion ($\beta$) consistency---using realistic recommendation scenarios constructed from MovieLens-100K. We evaluate four open-source models (Llama-3.1-8B, Mistral-7B, Qwen2.5-7B, Phi-3.5-mini) and find that all models violate every axiom at non-trivial rates, with IIA violations ranging from 30--59\% and Sen's $\alpha$ violations from 37--65\%. Transitivity (5--13\%) and Sen's $\beta$ (4--6\%) are the least violated axioms, while set-based choice axioms (IIA, $\alpha$) exhibit dramatically higher violation rates across all models, revealing a consistent pairwise--set-based dichotomy. Our results reveal a fundamental gap between the apparent fluency of LLM recommendations and their logical coherence, with implications for the trustworthiness of LLM-based recommendation systems.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Submission Number: 61