CLUES: A Benchmark for Learning Classifiers using Natural Language Explanations

Rakesh R Menon, Sayan Ghosh, Shashank Srivastava

TL;DR: We introduce a benchmark to learn classifiers, for structured datasets, from natural language explanations.

Abstract: Supervised learning has traditionally focused on inductive learning by observing labeled examples of a task. In contrast, humans have the ability to learn new concepts from language. Here, we explore training classifiers for structured data purely from language. We introduce CLUES, a benchmark for Classifier Learning Using natural language ExplanationS, consisting of a range of classification tasks over structured data along with language supervision in the form of explanations. CLUES consists of 36 real-world and 144 synthetic classification tasks. It contains crowdsourced explanations describing real-world tasks from multiple teachers and programmatically generated explanations for the synthetic tasks. We also develop ExEnt, an entailment-based model that learns classifiers using explanations. ExEnt generalizes up to 18% better (relative) on novel tasks than a baseline that does not use explanations.

Track: Non-Archival (will not appear in proceedings)

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/clues-a-benchmark-for-learning-classifiers/code)