PCNN: Probable-Class Nearest-Neighbor Explanations Improve Fine-Grained Image Classification Accuracy for AIs and Humans

Giang Nguyen, Valerie Chen, Mohammad Reza Taesiri, Anh Nguyen

Abstract:

Nearest neighbors (NN) are traditionally used to compute final decisions, e.g., in Support Vector Machines or k-NN classifiers, and to provide users with explanations for the model's decision.
In this paper, we show a novel utility of nearest neighbors: To improve predictions of a frozen, pretrained image classifier C.
We leverage an image comparator S that (1) compares the input image with NN images from the top-K most probable classes given by C; and (2) uses scores from S to weight the confidence scores of C to refine predictions.
Our method consistently improves fine-grained image classification accuracy on CUB-200, Cars-196, and Dogs-120.
Also, a human study finds that showing users our probable-class nearest neighbors (PCNN) reduces over-reliance on AI, thus improving their decision accuracy over prior work which only shows only the most-probable (top-1) class examples.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission: Finetune the Abstract, Add Sec. B3 and Revise wording.

Assigned Action Editor: ~Sivan_Sabato1

Submission Number: 2552