A Learnable Similarity Metric for Transfer Learning with Dynamics Mismatch

Ram Ananth Sreenivasan, Hyun-Rok Lee, Yeonjeong Jeong, Jongseong Jang, Dongsub Shim, Chi-Guhn Lee

Keywords: Reinforcement Learning, Transfer Learning

TL;DR: A similarity metric for augmenting Transfer learning with Dynamics Mismatch to avoid negative transfer

Abstract: When transferring knowledge from previously mastered source tasks to a new target task, the similarity between the source and target tasks can play a key role in whether such transfer is beneficial or harmful. In this paper, we develop an upper-bound of difference in action value function of source and target tasks with dynamics mismatch, and use the bound as a metric for dissimilarity between two tasks. The proposed metric does not require additional samples and adds little extra computation to the reinforcement learning algorithm for the target task. Also, the metric is highly portable so that it can be integrated into a wide range of algorithms. We showcase the effectiveness of the metric by incorporating it as a gatekeeper in the knowledge transfer step of transfer reinforcement learning algorithms. Numerical results on a suite of transfer learning scenarios demonstrate the benefits of preventing negative transfer in case of severe mismatch while accelerating learning otherwise

Submission Number: 2