Solving Truly Massive Budgeted Monotonic POMDPs with Oracle-Guided Meta-Reinforcement Learning

TMLR Paper6126 Authors

Abstract:

Many real-world decision problems, ranging from asset-maintenance scheduling to portfolio rebalancing, can be naturally modelled as budget-constrained multi-component monotonic Partially Observable Markov Decision Processes (POMDPs): each component’s latent state degrades stochastically until an expensive restorative action is taken, while all assets share a fixed intervention budget.
For a large numbers of assets, deriving an optimal policy for this joint POMDP is computationally intractable. To tackle this challenge, we prove that the value function of the associated belief-MDP is \emph{budget-concave}, which allows an efficient two-step approach to finding a near-optimal policy. First, we approximate the optimal cross-component budget split via a random-forest surrogate of each single-component value function. Second, we solve each resulting budget-constrained single-component POMDP with an oracle-guided meta-trained Proximal Policy Optimization (PPO) policy: value-iteration on the fully observable counterpart yields an oracle that shapes the PPO update and greatly accelerates learning. We validate our method through experiments in two disparate domains: (i) preventive maintenance for a large-scale building infrastructure containing 1,000 components, and (ii) portfolio risk management under debit-only loss-budget constraints, where each asset’s latent budget depletes with market losses and can only be replenished through costly recapitalization. Results show that our method consistently achieves longer component survival times and enhanced portfolio viability than both baseline heuristics and vanilla PPO. Furthermore, our approach maintains linear scalability in solution time with respect to the number of components.

Submission Length: Long submission (more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=olXzjN8xWh&referrer=%5BAuthor%20Console%5D(%2Fgroup%3Fid%3DTMLR%2FAuthors%23your-submissions)

Changes Since Last Submission:

**Changes Since Last Submission**

The previous version of this manuscript was desk rejected because the compiled PDF used a font different from the TMLR template default. Upon review, we identified that this issue was caused by including the line

```latex
\usepackage{times}
```

in the preamble, which overrides the default font specified by the TMLR style file.

In this resubmission, the `times` package has been completely removed, and the manuscript now relies entirely on the default font settings of the TMLR class file. No other stylistic or formatting changes have been made.

All mathematical content, figures, and tables remain identical to the previous submission. The only modification is the removal of the `times` package to ensure full compliance with the TMLR formatting requirements and to restore the correct default font.

Assigned Action Editor: ~Tongzheng_Ren1

Submission Number: 6126