Mind the Gap: LLMs, Competency Questions, and the Non-Technical User in the Humanities Domain

Claire McNamara, Lucy Hederman, Declan O'Sullivan

Keywords: Knowledge Graph Exploration, Competency Questions, Large Language Models, Non-Technical Users, Initial Exploration Problem

Abstract: Non-technical users often face a significant barrier when first attempting to explore complex knowledge graphs (KGs). We define this challenge as the Initial Exploration Problem, characterised by three interrelated barriers: ontology opacity, query incapacity, and scope uncertainty. This paper investigates how large language models (LLMs) can support domain experts in addressing this problem by automatically generating template-style competency questions (CQs) for the Virtual Record Treasury of Ireland (VRTI) KG. These templates are not user-facing themselves, but serve as scaffolding for creating curated questions (CuQs), expert-validated, natural language questions that help new users begin meaningfully exploring the graph. We evaluate two LLMs (GPT-4o and Gemini 2.0 Flash) across twelve prompt configurations varying in scope and framing, and assess question quality using both semantic similarity to expert-authored CQs and detailed expert review. Our findings highlight how prompt design influences LLM output, and underscore the value of combining automated generation with expert curation. Ultimately, we propose a practical pipeline to support the creation of exploratory entry points tailored to user needs, helping domain experts craft better questions, and helping users take their first steps into meaningful KG exploration.

Submission Number: 1