Deep Learning with Logged Bandit Feedback

Thorsten Joachims, Adith Swaminathan, Maarten de Rijke

Abstract: We propose a new output layer for deep neural networks that permits the use of logged contextual bandit feedback for training. Such contextual bandit feedback can be available in huge quantities (e.g., logs of search engines, recommender systems) at little cost, opening up a path for training deep networks on orders of magnitude more data. To this effect, we propose a Counterfactual Risk Minimization (CRM) approach for training deep networks using an equivariant empirical risk estimator with variance regularization, BanditNet, and show how the resulting objective can be decomposed in a way that allows Stochastic Gradient Descent (SGD) training. We empirically demonstrate the effectiveness of the method by showing how deep networks -- ResNets in particular -- can be trained for object recognition without conventionally labeled images.

TL;DR: The paper proposes a new output layer for deep networks that permits the use of logged contextual bandit feedback for training.

Keywords: Batch Learning from Bandit Feedback, Counterfactual Learning

Data: [CIFAR-10](https://paperswithcode.com/dataset/cifar-10)