Lessons learned in the study of representational alignment in physical reasoning
Track: short paper (up to 5 pages)
Keywords: physical reasoning, cognitive science, AI, cognitive AI benchmarking, scene understanding, intuitive physics, vision
Abstract: Recent developments allow AI systems to perform cognitively complex and rich tasks. At the same time, collecting human behavior at scale is more feasible than ever. This convergence of trends allows for the combined large-scale study of human and AI behavior in rich domains and tasks. Such experiments promise to provide better insight into the representations and strategies underlying both human and AI behavior. However, doing so in a way that does justice to both humans and AI systems is challenging. Here, we outline the key considerations and challenges we've faced in a benchmarking study investigating physical understanding across humans and AI systems and discuss how we've addressed them.
Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.
Submission Number: 20
Loading