Burhan at IslamicEval: Fact-Augmented and LLM-Driven Retrieval for Islamic QA

Mohammad Basheer, Watheq Mansour, Abdulhamid Touma, Ahmad Qadeib Alban

Keywords: Information Retrieval, Semantic Search, Ranking

Abstract: This paper presents our approach to the Qur'an and Hadith QA task in the IslamicEval 2025 Shared Task. Reliable retrieval requires both accuracy and context-aware answers from Qur’anic and Hadith text.To address this challenge, We combine semantic search with LLM-based re-ranking. To enhance alignment, we augment the corpus with LLM-extracted Islamic facts and paraphrased queries. An LLM-based binary classifier further verifies whether retrieved passages answer the questions. Results show improved accuracy and better alignment with user intent.

Submission Number: 7