Inferring cognitive strategies from groups of animals in natural environments

Ines Aitsahalia, Thomas L. Botch, Shijie Gu, Thomas O'Connell, Rebecca Siegel, Ralph E Peterson, Dmitry Batenkov, Emily L Mackevicius

Abstract: Understanding multi-agent behavior in natural environments requires world models that jointly capture agent interactions, environmental structure, and the cognitive strategies that shape collective dynamics. Existing approaches typically focus on isolated components — such as tracking, simulation, or inference — without integration into a unified pipeline for reasoning about relationships between behavior, environment, and cognition. Here, we introduce new behavioral data from groups of birds foraging in outdoor 3D environments, as well as an open-source framework for interpreting this data. The framework combines detection and tracking of multiple animals, semantic 3D environment reconstruction, multi-agent simulation, and graph neural network–based inference. By bringing together behavior, context, and predictive modeling, this paper lays the groundwork for investigating latent cognitive strategies across species and environments, and identifies key challenges to interpreting models of these systems.

Length: long paper (up to 8 pages)

Domain: data, methods

Author List Check: The author list is correctly ordered and I understand that additions and removals will not be allowed after the abstract submission deadline.

Anonymization Check: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and URLs that point to identifying information.

Submission Number: 25