Absorb What You Need: Accelerating Exploration via Valuable Knowledge Extraction

You Wu, Zhe Wu, Renye Yan, Pin Tao, Junliang Xing

Abstract: Leveraging external knowledge and extracting valuable insights are efficient human practices when handling various tasks. In contrast, current artificial intelligence lacks this capability. Recent research aims to teach Reinforcement Learning (RL) agents to incorporate external knowledge in the form of natural language to accelerate exploration. A common assumption in many of these approaches is that all introduced external knowledge is inherently valuable. To eliminate this assumption, we introduce the Knowledge Extraction Exploration Framework (KEEF). KEEF comprises two key components: 1) a knowledge extractor, designed to filter useful external knowledge based on three dimensions, task relevance, environment relevance, and achievement difficulty, through a prediction network and a policy network; and 2) a policy executor, which is a knowledge-conditioned network facilitating joint reasoning between the useful knowledge extracted by the knowledge extractor and the current state of the environment. In eight challenging sparse reward BabyAI environments, KEEF has consistently demonstrated superior sampling efficiency compared to knowledge-based and traditional RL methods.