Characterizing Human and Zero-Shot GPT-3.5 Object-Similarity Judgments

Anonymous

TL;DR: Characterizing the object-similarity judgment of GPT-3.5 with an interpretable embedding, then examining the utility of its responses in doing the same for humans.

Abstract:

Recent advancements in large language models' (LLMs) capabilities have yielded few-shot, human-comparable performance on a range of tasks. At the same time, researchers expend significant effort and resources gathering human annotations. At some point, LLMs may be able to perform some simple annotation tasks, but studies of LLM annotation accuracy and consistency are sparse. In this paper, we characterize OpenAI's ChatGPT's judgment on a behavioral task for implicit object categorization.
We characterize the embedding spaces of models trained on human vs.\ GPT responses and note similarities, but also systematic differences between them. We also find that augmenting a dataset of humans' responses with ChatGPT predictions causes models to diverge well before performance saturation.

Paper Type: short

Research Area: Interpretability and Analysis of Models for NLP

Contribution Types: Model analysis & interpretability, Approaches to low-resource settings, Data resources

Languages Studied: English