ComplexLogicalQA: A Comprehensive Benchmark for Complex Logical Question Answering over Knowledge Graphs

ACL ARR 2026 January Submission1736 Authors

Keywords: Knowledge Graph Question Answering, Benchmark, Dataset Construction

Abstract: Knowledge Graph Question Answering (KGQA) has evolved significantly with the advent of Large Language Models (LLMs). However, current benchmarks suffer from a severe structural bias. They are dominated by simple linear paths, while complex logical operations such as Disjunction (Union) and Nested Logic are noticeably scarce. Our quantitative analysis of mainstream KGQA datasets reveals that 94\% of questions are limited to chain-like reasoning, yet Union operations are completely absent. To bridge this gap, we propose ComplexLogicalQA, a comprehensive benchmark encompassing nine distinct Existential Positive First-Order Logic (EPFO) structures. We develop a novel logic-driven reverse-construction pipeline that leverages LLMs to verbalize sampled subgraphs, ensuring both structural complexity and linguistic diversity. Extensive evaluations across four paradigms reveal a significant reasoning illusion: while models excel at linear pattern matching, their performance collapses on Union and Nested logic. Further analysis identifies fundamental limitations in current paradigms, such as disjunction blindness in retrieval and premature pruning in agent-based search.

Paper Type: Long

Research Area: Question Answering

Research Area Keywords: Question Answering,Resources and Evaluation

Contribution Types: Model analysis & interpretability, Data resources, Data analysis

Languages Studied: English

Submission Number: 1736