ParallelPARC: A Scalable Pipeline for Generating Natural-Language Analogies

Anonymous

TL;DR: A Scalable Pipeline for Generating Natural-Language Analogies

Abstract:

Analogy-making is central to human cognition, allowing us to adapt to novel situations -- an ability that current AI systems still lack.
Most analogy datasets today focus on simple analogies (e.g., word analogies); datasets including complex types of analogies are typically manually curated and very small. We believe that this holds back progress in computational analogy.
In this work, we design a data generation pipeline, ParallelPARC (Parallel Paragraph Creator) leveraging state-of-the-art Large Language Models (LLMs) to create complex, paragraph-based analogies, as well as distractors, both simple and challenging.
We demonstrate our pipeline and create ProPara-Logy, a dataset of analogies between scientific processes. We publish a gold-set, validated by humans, and a silver-set, generated automatically. We test LLMs' and humans' analogy recognition in binary and multiple-choice settings, and found that humans outperform the best models (~13% gap) after a light supervision.
We demonstrate that our silver-set is useful for training models. Lastly, we show challenging distractors confuse LLMs, but not humans.
We hope our pipeline will encourage research in this emerging field.

Paper Type: long

Research Area: NLP Applications

Contribution Types: NLP engineering experiment, Data resources

Languages Studied: English