Bridging the Gap: Converting Human Advice into Imagined Examples

Eric Yeh, Melinda Gervasio, Daniel Sanchez, Matthhew Crossley, Karen L. Myers

Abstract:

Advice is a powerful tool for learning. But advice also presents the challenge of bridging the gap
between the high-level representations that easily capture human advice and the low-level repre-
sentations that systems must operate with using that advice. Drawing inspiration from studies on
human motor skills and memory systems, we present an approach that converts human advice into
synthetic or imagined training experiences, serving to scaffold the low-level representations of sim-
ple, reactive learning systems such as reinforcement learners. Research on using mental imagery
and directed attention in motor and perceptual skills motivates our approach. We introduce the con-
cept of a cognitive advice template for generating scripted, synthetic experiences and use saliency
masking to further conceal irrelevant portions of training observations. We present experimental
results for a deep reinforcement learning agent in a Minecraft-based game environment that show
how such synthetic experiences improve performance, enabling the agent to achieve faster learning
and higher rates of success.