A Closed-Loop System for Improving Annotation Quality and Efficiency

Anonymous

Keywords: annotation, metrics, AB testing, human-in-the-loop, segmentation, analytics

TL;DR: We describe a general experimental pipeline to improve the quality and efficiency of human-in-the-loop data annotation.

Abstract: We present a general system and approach to improve the quality and efficiency of interactive annotation. A specific use case based on instance segmentation of vehicles for autonomous driving is used as an illustration. Via incremental AB testing and a custom analytics pipeline, we show how to optimize human-ML interaction to systematically improve annotation efficiency, and address the shortcomings of ML models.