ThinkDrill at IslamicEval 2025 Shared Task: LLM Hybrid Approach for Qur’an and Hadith Question Answering

Eman Elrefai, Toka Khaled, Ahmed Soliman

Keywords: Qur'an question answering, Hadith retrieval, Arabic NLP, hybrid retrieval, embeddings, large language models, triplet loss, IslamicEval 2025, Reranker

TL;DR: We propose a hybrid multi-pipeline system combining fine-tuned embeddings, keyword matching, and LLM-guided retrieval to improve Quran and Hadith question answering in Arabic.

Abstract: This paper presents our approach to Subtask 2 of IslamicEval 2025, a shared task that involves retrieving relevant passages from Quranic verses and Sahih Bukhari hadiths to answer Modern Standard Arabic (MSA) questions. We developed a multi-pipeline hybrid system that combines three complementary approaches: fine-tuned embedding models using triplet loss, keyword-based fuzzy matching, and large language model guided retrieval. Our system achieved MAP_@10 of 0.2296, MAP_Q@5 of 0.2623, and MAP_H@5 of 0.215 in the test set, demonstrating the effectiveness of combining multiple retrieval strategies for Arabic religious text question answering.

Submission Number: 6