Knowledge Acquisition Passage Retrieval: Corpus, Ranking Models, and Evaluation Resources

Artemis Çapari, Hosein Azarbonyad, Georgios Tsatsaronis, Zubair Afzal, Judson Dunham, Jaap Kamps

Abstract: Knowledge acquisition passage retrieval is a task that captures search in a learning or educational setting, where users seek to find key educational information within their field of interest. Traditional relevance assessments used in ad-hoc retrieval tasks tend to focus on topical relevance, often overlooking other factors such as the “informativeness” of the retrieved educational content in relation to the user’s knowledge acquisition needs. This paper presents a new test collection for the knowledge acquisition passage retrieval (KAPR) task, constructed using the data and production systems of a large academic publisher containing: First, a set of search requests covering key educational topics/concepts across different science domains. Second, a large corpus of passages extracted from review (survey) articles published in over 2, 700 journals as well as the content of 43, 000 books published in a wide range of science domains. Third, relevance assessments on both topical relevance as well as informativeness, reflecting the task-specific relevance. This resource enables direct evaluation of the user’s utility of the retrieved content and provides a comparative analysis with traditional topical relevance. Our findings indicate a strong correlation between relevance and informativeness, although the distribution of these labels varies per domain.