A Simple Heuristic for Controlling Human Workload in Learning to Defer

Andrew Ponomarev

Abstract: In many cases, machine learning model is used not autonomously, but as a part of some larger system that may include human experts. Learning to defer technique allows to train models that can take into account error probabilities of both machine learning model and human expert and route samples accordingly in order to maximize overall accuracy of the system. However, most of the learning to defer methods don’t allow constraining the deferral fraction, which is important, as the number of human experts and their capacity are usually limited. The paper proposes and explores a simple yet effective heuristic technique allowing to impose constraints on the fraction of samples deferred to an expert, thereby, helping to balance accuracy and coverage metrics. The technique can be used in conjunction with many existing learning to defer and rejection learning methods; it is evaluated using three popular learning to defer techniques and two datasets — a synthetic and a real-life, collected using crowdsourcing.