Hence, Socrates is mortal: A Benchmark for Natural Language Syllogistic Reasoning

Anonymous

Abstract: Syllogistic reasoning, a typical form of deductive reasoning, is a critical capability widely required in natural language understanding tasks, such as text entailment and question answering. To better facilitate research on syllogistic reasoning, we develop a benchmark called SylloBase that differs from existing syllogistic datasets in three aspects: (1) Covering a complete taxonomy of syllogism reasoning patterns; (2) Containing both automatically and manually constructed samples; and (3) Involving both the generation and understanding tasks. We automatically construct 250k template-based syllogism samples by mining syllogism patterns from Wikidata and ConceptNet. To improve our dataset's naturalness and challenge, we manually rewrite 1,000 samples from template-based data by adding distracting noise and paraphrasing as the test set. State-of-the-art pre-trained language models can achieve the best generation ROUGE-L of 38.06 by BART and the best multi-choice accuracy of 77% by RoBERTa on SylloBase, which indicates the great challenge of learning diverse syllogistic reasoning types on SylloBase.

Paper Type: long