Help Me Explore: Minimal Social Interventions for Graph-Based Autotelic Agents

Ahmed Akakzia, Olivier Serris, Olivier Sigaud, Cédric Colas

Keywords: Deep reinforcement learning, intrinsic motivations, autonomous learning, social learning

Abstract: In the quest for autonomous agents learning open-ended repertoires of skills, most works take a Piagetian perspective: learning trajectories are the results of interactions between developmental agents and their physical environment. The Vygotskian perspective, on the other hand, emphasizes the centrality of the socio-cultural environment: higher cognitive functions emerge from transmissions of socio-cultural processes internalized by the agent. This paper acknowledges these two perspectives and presents GANGSTR, a hybrid agent engaging in both individual and social goal-directed exploration. In a 5-block manipulation domain, GANGSTR discovers and learns to master tens of thousands of configurations. In individual phases, the agent engages in autotelic learning; it generates, pursues and makes progress towards its own goals. To this end, it builds a graph to represent the network of discovered configuration and to navigate between them. In social phases, a simulated social partner suggests goal configurations at the frontier of the agent’s current capabilities. This paper makes two contributions: 1) a minimal social interaction protocol called Help Me Explore (HME); 2) GANGSTR, a graph-based autotelic agent. As this paper shows, coupling individual and social exploration enables the GANGSTR agent to discover and master the most complex configurations (e.g. stacks of 5 blocks) with only minimal intervention.

One-sentence Summary: We propose an autotelic agent that discovers and masters thousands of semantic goal configurations and a social interaction protocol where a social partner drives the agent towards its zone of proximal development.