Shaping embodied agent behavior with activity-context priors from egocentric video

Tushar Nagarajan, Kristen Grauman

Keywords: embodied AI, learning from passive videos, visual semantic planning, interaction exploration, object interaction, egocentric video, human activity understanding, computer vision

TL;DR: We learn from real-world egocentric video what objects together enable activities (e.g., a knife and cutting board brought together with a tomato are conducive to cutting), and use this to improve training virtual household embodied AI agents.

Abstract: Complex physical tasks entail a sequence of object interactions, each with its own preconditions -- which can be difficult for robotic agents to learn efficiently solely through their own experience. We introduce an approach to discover activity-context priors from in-the-wild egocentric video captured with human worn cameras. For a given object, an activity-context prior represents the set of other compatible objects that are required for activities to succeed (e.g., a knife and cutting board brought together with a tomato are conducive to cutting). We encode our video-based prior as an auxiliary reward function that encourages an agent to bring compatible objects together before attempting an interaction. In this way, our model translates everyday human experience into embodied agent skills. We demonstrate our idea using egocentric EPIC-Kitchens video of people performing unscripted kitchen activities to benefit virtual household robotic agents performing various complex tasks in AI2-iTHOR, significantly accelerating agent learning.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.