Towards a More Rigorous Science of Blindspot Discovery in Image Classification Models
Authors that are also TMLR Expert Reviewers: ~Gregory_Plumb2
Abstract: A growing body of work studies Blindspot Discovery Methods ("BDM"s): methods that use an image embedding to find semantically meaningful (i.e., united by a human-understandable concept) subsets of the data where an image classifier performs significantly worse. Motivated by observed gaps in prior work, we introduce a new framework for evaluating BDMs, SpotCheck, that uses synthetic image datasets to train models with known blindspots and a new BDM, PlaneSpot, that uses a 2D image representation. We use SpotCheck to run controlled experiments that identify factors that influence BDM performance (e.g., the number of blindspots in a model, or features used to define the blindspot) and show that PlaneSpot is competitive with and in many cases outperforms existing BDMs. Importantly, we validate these findings by designing additional experiments that use real image data from MS-COCO, a large image benchmark dataset. Our findings suggest several promising directions for future work on BDM design and evaluation. Overall, we hope that the methodology and analyses presented in this work will help facilitate a more rigorous science of blindspot discovery.
Certifications: Expert Certification
Submission Length: Regular submission (no more than 12 pages of main content)
Changes Since Last Submission: Revision #1: See highlighted lines.
Revision #2: Added results from additional real data experiments (using OpenImages) in Appendix H.
Revision #3 (Camera Ready): Added links to code, authors and affiliations, and acknowledgements.
Code: https://github.com/njohnson99/spotcheck
Assigned Action Editor: ~Krishnamurthy_Dvijotham2
License: Creative Commons Attribution 4.0 International (CC BY 4.0)
Submission Number: 1056
Loading