Uncertainty-aware off policy learning

Xiaoying Zhang, Junpu Chen, Hongning Wang, Hong Xie, Hang Li

Keywords: off-policy learning, uncertainty

TL;DR: We consider the estimation uncertainty of logging policy, and proposed a new estimator for improved off-policy learning by controlling the effect of inaccurate estimation of logging policy.

Abstract:

Off-policy learning, referring to the procedure of policy optimization with access only to logged feedback data, has shown importance in various real-world applications, such as search engines, recommender systems, etc. While the ground-truth logging policy, which generates the logged data, is usually unknown, previous work directly takes its estimated value in off-policy learning, resulting in a biased estimator. This estimator has both high bias and variance on samples with small and inaccurate estimated logging probabilities.
In this work, we explicitly model the uncertainty in the estimated logging policy and propose a novel \underline{U}ncertainty-aware \underline{I}nverse \underline{P}ropensity \underline{S}core estimator (UIPS) for improved off-policy learning. Experiment results on synthetic and three real-world recommendation datasets demonstrate the advantageous sample efficiency of the proposed UIPS estimator.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Reinforcement Learning (eg, decision and control, planning, hierarchical RL, robotics)