Generative Example-Driven Error Mining

Constantine Nakos, Walker Demel, Kenneth Forbus

Keywords: natural language understanding, natural language generation, error mining

Abstract: Hand-curated natural language (NL) systems use explicit linguistic knowledge to understand and generate text. However, maintaining this knowledge can be a challenge. Errors abound in linguistic resources of any size, and tracking them down requires expertise that could better be applied elsewhere. In this work, we introduce Generative Example-Driven Error Mining (GEDEM), a technique for discovering errors in hand-curated NL systems capable of generation. GEDEM generates example sentences based on the system's linguistic knowledge and asks the user to identify any sentences that are ungrammatical or which differ semantically from the others. It then uses these judgments to diagnose the errors in the system's linguistic knowledge that caused the bugs in the generated examples. We show the utility of our framework by applying it to Companions Natural Language Understanding (CNLU), a knowledge-rich semantic parser with a generation component.

Paper Track: Technical paper

Submission Number: 25