Fast Approximate Dynamic Programming for Infinite-Horizon Markov Decision ProcessesDownload PDF

21 May 2021, 20:44 (modified: 21 Oct 2021, 10:59)NeurIPS 2021 PosterReaders: Everyone
Keywords: Stochastic optimal control, value iteration, input-affine systems, Fenchel duality, computational complexity.
Abstract: In this study, we consider the infinite-horizon, discounted cost, optimal control of stochastic nonlinear systems with separable cost and constraints in the state and input variables. Using the linear-time Legendre transform, we propose a novel numerical scheme for implementation of the corresponding value iteration (VI) algorithm in the conjugate domain. Detailed analyses of the convergence, time complexity, and error of the proposed algorithm are provided. In particular, with a discretization of size $X$ and $U$ for the state and input spaces, respectively, the proposed approach reduces the time complexity of each iteration in the VI algorithm from $O(XU)$ to $O(X+U)$, by replacing the minimization operation in the primal domain with a simple addition in the conjugate domain.
Supplementary Material: pdf
Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.
Code: https://github.com/AminKolarijani/ConjVI
18 Replies

Loading