A Planning Algorithm for Predictive State Representations

Masoumeh T. Izadi, Doina Precup

Abstract: We address the problem of optimally controlling stochastic environments that are partially observable. The standard method for tackling such problems is to dene and solve a Partially Observable Markov Decision Process (POMDP). However, it is well known that exactly solving POMDPs is very costly computationally. Recently, Littman, Sutton and Singh (2002) have proposed an alternative representation of partially observable environments, called predictive state representations (PSRs). PSRs are grounded in the sequence of actions and observations of the agent, and hence relate the state representation directly to the agent’s experience. In this paper, we present a policy iteration algorithm for nding policies using PSRs. In preliminary experiments, our algorithm produced good solutions.