Definitions Matter: Guiding GPT for Multi-label Classification

Youri Peskine, Damir Korenčić, Ivan Grubisic, Paolo Papotti, Raphael Troncy, Paolo Rosso

Submission Type: Regular Short Paper

Submission Track: Language Modeling and Analysis of Language Models

Submission Track 2: NLP Applications

Keywords: GPT-3, zero-shot classification, LLM

TL;DR: We analyze the importance of label definition in GPT zero-shot labeling and introduce a new method to generate definition from examples

Abstract: Large language models have recently risen in popularity due to their ability to perform many natural language tasks without requiring any fine-tuning. In this work, we focus on two novel ideas: (1) generating definitions from examples and using them for zero-shot classification, and (2) investigating how an LLM makes use of the definitions. We thoroughly analyze the performance of GPT-3 model for fine-grained multi-label conspiracy theory classification of tweets using zero-shot labeling. In doing so, we asses how to improve the labeling by providing minimal but meaningful context in the form of the definitions of the labels. We compare descriptive noun phrases, human-crafted definitions, introduce a new method to help the model generate definitions from examples, and propose a method to evaluate GPT-3's understanding of the definitions. We demonstrate that improving definitions of class labels has a direct consequence on the downstream classification results.

Submission Number: 3394