Decision-aware Training of Spatiotemporal Forecasting Models to Select a Top-K Subset of Sites for Intervention

Kyle Heuton, Frederick Muench, Shikhar Shrestha, Thomas J. Stopka, Michael C Hughes

TL;DR: We develop methods for selecting exactly K sites out of many with probabilistic spatiotemporal forecasting models.

Abstract:

Optimal allocation of scarce resources is a common problem for decision makers faced with choosing a limited number of locations for intervention. Spatiotemporal prediction models could make such decisions data-driven.
A recent performance metric called fraction of best
possible reach (BPR) measures the impact of using a model’s recommended size K subset of sites compared to the best possible top-K in hindsight. We tackle two open problems related to BPR. First, we explore *how to rank* all sites numerically given a probabilistic model that predicts event counts jointly across sites. Ranking via the per-site mean is suboptimal for BPR. Instead, we offer a better ranking for BPR backed by decision theory. Second, we explore*how to train* a probabilistic model's parameters to maximize BPR.
Discrete selection of K sites implies all-zero parameter gradients which prevent standard gradient training. We overcome this barrier via advances in perturbed optimizers. We further suggest a training objective that combines likelihood with a BPR constraint to deliver high-quality top-K rankings as well as good forecasts for all sites. We demonstrate our approach on two where-to-intervene applications: mitigating opioid-related fatal overdoses for public health and monitoring endangered wildlife.

Lay Summary:

The opioid crisis is widespread, but public health agencies have limited budgets and must strategically choose only some locations to intervene in. Standard machine learning approaches are created to accurately predict the number of overdoses in each location equally. We find that this approach is not always best for identifying the few locations that need intervention the most.

We develop an approach that shows how to use machine learning models with the goal of making better decisions with limited resources. While the traditional approach optimizes for accurate predictions of overdoses, we also show a second approach that is instead trained for optimizing decision quality, or accurately identifying the top places in which to intervene. Finally, we explore a third approach that balances both optimizing for accuracy of overdoses and decision quality called “Decision-aware.”

We show how using this “Decision-aware” approach can lead to more accurate allocation of resources where they are needed most, not only within the context of opioid interventions but also within other contexts, such as placing cameras to observe endangered birds. We hope that this research allows those who make decisions based on model forecasts to better maximize their impact.

Application-Driven Machine Learning: This submission is on Application-Driven Machine Learning.

Link To Code: https://github.com/tufts-ml/decision-aware-topk

Primary Area: Probabilistic Methods

Keywords: spatiotemporal forecasting, top K, decision theory, probabilistic models

Submission Number: 8826