Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy EvaluationDownload PDF

Published: 11 Oct 2021, Last Modified: 23 May 2023NeurIPS 2021 Datasets and Benchmarks Track (Round 2)Readers: Everyone
Keywords: off-policy evaluation, real-world dataset, open-source software, benchmark experiments, offline contextual bandits
TL;DR: Large-scale public real dataset and open-source software to enable realistic and reproducible experiments and implementations of off-policy evaluation
Abstract: \textit{Off-policy evaluation} (OPE) aims to estimate the performance of hypothetical policies using data generated by a different policy. Because of its huge potential impact in practice, there has been growing research interest in this field. There is, however, no real-world public dataset that enables the evaluation of OPE, making its experimental studies unrealistic and irreproducible. With the goal of enabling realistic and reproducible OPE research, we present \textit{Open Bandit Dataset}, a public logged bandit dataset collected on a large-scale fashion e-commerce platform, ZOZOTOWN. Our dataset is unique in that it contains a set of \textit{multiple} logged bandit datasets collected by running different policies on the same platform. This enables experimental comparisons of different OPE estimators for the first time. We also develop Python software called \textit{Open Bandit Pipeline} to streamline and standardize the implementation of batch bandit algorithms and OPE. Our open data and software will contribute to fair and transparent OPE research and help the community identify fruitful research directions. We provide extensive benchmark experiments of existing OPE estimators using our dataset and software. The results open up essential challenges and new avenues for future OPE research.
Supplementary Material: pdf
URL: Public Real-World Dataset: / Open-Source Software (Open Bandit Pipeline):
Contribution Process Agreement: Yes
Dataset Url:
License: The dataset is licensed under CC BY 4.0.
Author Statement: Yes
11 Replies