NUR at IslamicEval 2025 Shared Task: Retrieval-Augmented LLMs for Qur'an and Hadith QA

Serag Amin, Ranwa Aly, Yara Allam, Yomna Eid, Ensaf Hussein Mohamed

Keywords: Question Answering, Islamic Content, Information Retrieval, Transformers, BERT, LLM, Gemini

TL;DR: We present a contribution to the IslamicEval Subtask 2 comparing BERT-based transformers and Gemini 2.5 Flash.

Abstract: In this paper, we present our contribution to the IslamicEval 2025 shared task. More specifically, we address subtask 2, which is a passage retrieval (PR) system for Qur'an and Hadith, the two central bodies of text in Islam. Basing off of a fine-tuned BERT-based sentence transformer retrieval model, we explore several approaches, including pipelined fine-tuning of cross-encoders, as well as using a state-of-the-art LLM for reranking of relevant passages, and identification of zero-answer questions. Our best-performing system achieves a MAP@10 of 0.1809, MAP_Q@5 of 0.2334, and MAP_H@5 of 0.1923 on the test set.

Submission Number: 2