Distributionally Robust Policy Learning under Concept Drifts
Keywords: distributionally robust optimization, offline policy learning, concept drift, bandit learning, reinforcement learning.
Abstract: Distributionally robust policy learning aims to find a policy that performs well
under the worst-case distributional shift, and yet most existing methods for
robust policy learning consider the worst-case {\em joint} distribution of
the covariate and the outcome. The joint-modeling strategy can be unnecessarily conservative
when we have more information on the source of distributional shifts. This paper studies
a more nuanced problem --- robust policy learning under the \emph{concept drift},
when only the conditional relationship between the outcome and the covariate changes.
To this end, we first provide a doubly-robust estimator for evaluating
the worst-case average reward of a given policy under a set of perturbed conditional distributions.
We show that the policy value estimator enjoys asymptotic normality even if the nuisance parameters
are estimated with a slower-than-root-$n$ rate.
We then propose a learning algorithm that outputs the policy maximizing the
estimated policy value within a given policy class $\Pi$, and show
that the sub-optimality gap of the proposed algorithm is of the order
$\kappa(\Pi)n^{-1/2}$, with $\kappa(\Pi)$ is the entropy integral of $\Pi$ under the Hamming distance
and $n$ is the sample size. A matching lower bound is provided to show the optimality of the rate.
The proposed methods are implemented and evaluated in numerical studies,
demonstrating substantial improvement compared with existing benchmarks.
Primary Area: causal reasoning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 12853
Loading