Towards Automatic Discovery and Explanation of Differences Between Vision Models

Nam Hyeon-Woo, Tae-Hyun Oh, Zeynep Akata, Stephan Alaniz

Keywords: Explainability, Agent AI, Synthetic Data

TL;DR: We present an automatic framework for generating natural language explanations of comparison of two vision models and introduce three evaluation metrics and methods.

Abstract:

Researchers and developers often compare state-of-the-art and newly developed models beyond benchmark scores, using techniques such as visualizations, case-by-case analyses, and qualitative evaluations. Such analyses provide deeper insights into model behaviors and often motivate the development of improved models and the establishment of new benchmarks. However, identifying strengths and weaknesses typically requires extensive human effort, consuming a significant amount of time and resources. To address this challenge, we explore the automatic generation of natural language explanations that describe the performance differences between two models. We introduce three evaluation metrics for explanations: Completeness for correctness and overall informativeness, Density for token-level informativeness, and Token Length for the verbosity of explanations. Building on these metrics, we propose three explanation generation methods: Raw Differences, which enumerates all performance differences; Summarization, which condenses them into concise summaries;
and Optimization, which optimizes explanations for both informativeness and conciseness. We evaluate our framework on CMNIST, CLEVR, and CelebA, showing that Optimization effectively uncovers model differences and biases in natural language. For reproducibility, we will release the code and data.

Primary Area: interpretability and explainable AI

Submission Number: 11113