AI-enhanced semantic feature norms for 786 concepts

Siddharth Suresh, Kushin Mukherjee, Tyler Giallanza, Xizheng Yu, Mia Patil, Jonathan D. Cohen, Timothy T. Rogers

Keywords: large language models, human-in-the-loop, human-AI alignment, interpretability

TL;DR: The paper presents an AI-enhanced semantic norm dataset that fuses human feature norms with LLM responses (verified by GPT-4) for 786 concepts, yielding higher feature density and improved prediction of human similarity judgments.

Abstract:

Semantic feature norms have been foundational in the study of human conceptual
knowledge, yet traditional methods face trade-offs between concept/feature
coverage and verifiability of quality due to the labor-intensive nature of norming
studies. Here, we introduce a novel approach that augments a dataset of human-generated
feature norms with responses from large language models (LLMs) while
verifying the quality of norms against reliable human judgments. We find that our
AI-enhanced feature norm dataset shows much higher feature density and overlap
among concepts while outperforming a comparable human-only norm dataset and
word-embedding models in predicting people’s semantic similarity judgments.
Taken together, we demonstrate that human conceptual knowledge is richer than
captured in previous norm datasets and show that, with proper validation, LLMs
can serve as powerful tools for cognitive science research.

Submission Type: Long Paper (9 Pages)

Archival Option: This is a non-archival submission

Presentation Venue Preference: ICLR 2025

Submission Number: 68