Loosely Consistent Emphatic Temporal-Difference Learning

Jiamin He, Fengdi Che, Yi Wan, A. Rupam Mahmood

Keywords: off-policy policy evaluation, function approximation

TL;DR: We introduced the concept of loose consistency and proposed the first practical, loosely consistent off-policy TD algorithm under general linear function approximation.

Abstract: There has been significant interest in searching for off-policy Temporal-Difference (TD) algorithms that find the same solution that would have been obtained in the on-policy regime. An important property of such algorithms is that their expected update has the same fixed point as that of On-policy TD($\lambda$), which we call \emph{loose consistency}. Notably, Full-IS-TD($\lambda$) is the only existing loosely consistent method under general linear function approximation but, unfortunately, has a high variance and is scarcely practical. This notorious high variance issue motivates the introduction of ETD($\lambda$), which tames down the variance but has a biased fixed point. Inspired by these two methods, we propose a new loosely consistent algorithm called \emph{Average Emphatic TD} (AETD($\lambda$)) with a transient bias, which strikes a balance between bias and variance. Further, we unify AETD($\lambda$) with existing methods and obtain a new family of loosely consistent algorithms called \emph{Loosely Consistent Emphatic TD} (LC-ETD($\lambda$, $\beta$, $\nu$)), which can control a smooth bias-variance trade-off by varying the speed at which the transient bias fades. Through experiments on illustrative examples, we show the effectiveness and practicality of LC-ETD($\lambda$, $\beta$, $\nu$).