Escaping Stochastic Traps with Aleatoric Mapping Agents

Augustine N. Mavor-Parker, Kimberly A Young, Caswell Barry, Lewis Griffin

Keywords: Curiosity, Neuroscience, Acetylcholine, Uncertainty, Reinforcement learning, Intrinsic Rewards

Abstract: When extrinsic rewards are sparse, artificial agents struggle to explore an environment. Curiosity, implemented as an intrinsic reward for prediction errors, can improve exploration but fails when faced with action-dependent noise sources. We present aleatoric mapping agents (AMAs), a neuroscience inspired solution modeled on the cholinergic system of the mammalian brain. AMAs aim to explicitly ascertain which dynamics of the environment are unpredictable, regardless of whether those dynamics are induced by the actions of the agent. This is achieved by generating separate forward predictions for the mean and aleatoric uncertainty of future states with reducing intrinsic rewards for those states that are unpredictable. We show AMAs are able to effectively circumvent action-dependent stochastic traps that immobilise conventional curiosity driven agents.

One-sentence Summary: Avoiding distractions within curiosity driven learning using neuroscience inspired uncertainty computations.

Community Implementations: [ 4 code implementations](https://www.catalyzex.com/paper/arxiv:2102.04399/code)