MIRAGE: Auditing Anti-Muslim Bias in Frontier LLMs Across Reasoning, Agentic, and Time-Coupled Conditions

Noor Islam S. Mohammad, TAMIM SHEIKH

Keywords: Algorithmic Bias, Religious Bias, Large Language Models, Agentic AI, Chain-of-Thought, Fairness

TL;DR: MIRAGE shows anti-Muslim bias in frontier LLMs increases when models use chain-of-thought reasoning, agentic decision-making, or retrieval-augmented generation, and that prompt-based mitigation methods do not generalize across these settings.

Abstract: Five years after the discovery of persistent anti-Muslim bias in large language models, most evaluations remain confined to single-turn prompt completion, a setting that no longer reflects how frontier LLMs are deployed. We introduce MIRAGE (Muslim-Identity Reasoning and Agentic Generation Evaluation), a benchmark of 1,200 prompts spanning three deployment-realistic conditions: direct completion, chain-of-thought reasoning, and simulated agentic decision-making across content moderation, lending triage, refugee claim summarization, and hiring screens. Across six frontier models, we find that (i) chain-of-thought reasoning amplifies rather than suppresses Muslim-violence associations by 12-34\% relative to direct completion; (ii) agentic decisions exhibit a 9-22 percentage-point asymmetry between Muslim and matched non-Muslim cases on identical evidence; and (iii) bias is sharply time-coupled to retrieved news context, increasing 18-27\% under recent-conflict retrieval. Existing prompt-based mitigations transfer poorly across our three conditions, suppressing direct-completion bias while leaving agentic asymmetry largely intact. We release MIRAGE and an open evaluation harness to support targeted mitigation research. https://pmlrbd.github.io/mirage1/

Track: Track 2: ML Research by Muslim Authors

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Non Archival Confirmation: I understand that submissions to MusIML are non-archival and can be submitted to other venues.

Submission Number: 88