Incremental Uncertainty-aware Performance Monitoring with Labeling Intervention

Alexander Koebler, Thomas Decker, Ingo Thon, Volker Tresp, Florian Buettner

Keywords: Model Monitoring, Uncertainty, Temporal Distribution Shifts, Optimal Transport

TL;DR: Incremental Uncertainty-aware Performance Monitoring (IUPM) is a novel label-free method that estimates model performance by modeling time-dependent shifts, allowing for uncertainty quantification and active labeling.

Abstract: We study the problem of monitoring machine learning models under temporal distribution shifts, where circumstances change gradually over time, often leading to unnoticed yet significant declines in accuracy. We propose Incremental Uncertainty-aware Performance Monitoring (IUPM), a novel label-free method that estimates model performance by modeling time-dependent shifts using optimal transport. IUPM also quantifies uncertainty in performance estimates and introduces an active labeling strategy to reduce this uncertainty. We further showcase the benefits of IUPM on different datasets and simulated temporal shifts over existing baselines.

Submission Number: 67