Implicit Bayesian Markov Decision Process for Resource Efficient Decisions in Drug Discovery

Tianchi Chen, Jan Bíma, Otto Ritter, Sean L. Wu, Bo Yuan, Xiang Yu

Keywords: Bayesian Markov Decision Process, ensemble approach, similarity-based metric, sequential decision making

TL;DR: We propose an Implicit Bayesian Markov Decision Process (IB-MDP) algorithm, which effectively balances the trade-offs between uncertainty reduction and resource allocation, resulting in improved decision-making in drug discovery.

Abstract: In drug discovery, researchers make sequential decisions to schedule experiments, aiming to maximize the probability of success towards drug candidates while simultaneously minimizing expected costs. However, such tasks pose significant challenges due to complex trade-offs between uncertainty reduction and allocation of constrained resources in a high-dimensional state-action space. Traditional methods based on simple rule-based heuristics or domain expertise often result in either inefficient resource utilization due to risk aversion or missed opportunities due to reckless decisions. To address these challenges, we developed an Implicit Bayesian Markov Decision Process (IB-MDP) algorithm that constructs an explicit MDP model of the environment’s dynamics by integrating historical data through a similarity-based metric and enables effective planning by simulating future states and actions. To enhance the robustness of the decision-making process, the IB-MDP also incorporates an ensemble approach that recommends maximum likelihood actions to effectively balance the dual objectives of reducing state uncertainty and optimizing expected costs. Our experimental results demonstrate that the IB-MDP algorithm offers significant improvements over traditional rule-based methods by identifying optimal decisions that ensure more efficient use of resources in drug discovery.

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3142