Principles of Animal Cognition for LLM Evaluations: A Case Study on Transitive Inference

Sunayana Rane, Cyrus Kirkman, Amanda Royka, Graham Todd, Ryan Law, Jacob Gates Foster, Erica Cartmill

Keywords: behavior, concept learning, animal cognition, LLMs

TL;DR: What animal cognition research can teach us about designing better experiments probing LLM behavior

Abstract: It has become increasingly challenging to understand and evaluate LLM capabilities as these models exhibit a broader range of behaviors. We respond to this challenge by taking inspiration from another field which has developed ideas, practices, and paradigms for probing the behavior of complex intelligent systems: animal cognition. We present five core principles from animal cognition, and explain how they provide invaluable guidance for understanding LLM abilities and behavior. We ground these principles in an empirical case study, and show how they can provide a richer contextual picture of one particular reasoning capability: transitive inference.

Submission Number: 67