Deep Recurrent Optimal Stopping

NIRANJAN DAMERA VENKATA, Chiranjib Bhattacharyya

Keywords: optimal stopping, recurrent neural networks, probabilistic graphical models, policy gradient methods

TL;DR: We introduce an optimal stopping policy gradient algorithm (OSPG) that can leverage RNNs effectively in non-Markovian settings. OSPG works offline and eliminates expensive Monte Carlo policy rollouts.

Abstract: Deep neural networks (DNNs) have recently emerged as a powerful paradigm for solving Markovian optimal stopping problems. However, a ready extension of DNN-based methods to non-Markovian settings requires significant state and parameter space expansion, manifesting the curse of dimensionality. Further, efficient state-space transformations permitting Markovian approximations, such as those afforded by recurrent neural networks (RNNs), are either structurally infeasible or are confounded by the curse of non-Markovianity. Considering these issues, we introduce, for the first time, an optimal stopping policy gradient algorithm (OSPG) that can leverage RNNs effectively in non-Markovian settings by implicitly optimizing value functions without recursion, mitigating the curse of non-Markovianity. The OSPG algorithm is derived from an inference procedure on a novel Bayesian network representation of discrete-time non-Markovian optimal stopping trajectories and, as a consequence, yields an offline policy gradient algorithm that eliminates expensive Monte Carlo policy rollouts.

Submission Number: 11859