Dynamic Human AI Collaboration

Parth Pahwa, Kabir Thakur, Francois Buet-Golfouse

Keywords: Human AI collaboration, Deferral System, Ensemble, Bayesian Regression

TL;DR: Framework to supplement algorithmic decision making with expert opinions

Abstract: Domain experts possess valuable knowledge and insights that can help improve the accuracy and relevance of the machine learning (ML) models. By incorporating expert opinions, the models can capture important nuances and factors that may not be captured by data-driven methods alone. The integration of machine learning models with human experts has become increasingly common in real-world applications. In this paper, we propose a Bayesian framework for human-in-the-loop pipelines. We consider the scenario where the final decision is an amalgamation of algorithm and expert opinions and deferral systems are a special case. We finally show that updating expert opinion priors with information sharing between experts is key to achieving superior performance.