Evaluating Systemic Error Detection Methods using Synthetic ImagesDownload PDF

Published: 21 Jul 2022, Last Modified: 17 Sept 2023SCIS 2022 PosterReaders: Everyone
TL;DR: A new method and evaluation for finding systemic errors in image classifiers
Abstract: We introduce SpotCheck, a framework for generating synthetic datasets to use for evaluating methods for discovering blindspots (i.e., systemic errors) in image classifiers. We use SpotCheck to run controlled studies of how various factors influence the performance of blindspot discovery methods. Our experiments reveal several shortcomings of existing methods, such as relatively poor performance in settings with multiple blindspots and sensitivity to hyperparameters. Further, we find that a method based on dimensionality reduction, PlaneSpot, is competitive with existing methods, which has promising implications for the development of interactive tools.
Confirmation: Yes
Community Implementations: [![CatalyzeX](/images/catalyzex_icon.svg) 1 code implementation](https://www.catalyzex.com/paper/arxiv:2207.04104/code)
0 Replies