Interpretable Concept Discovery and Learning from Pretrained Vision-Language Models

Yuan Zang, Tian Yun, Hao Tan, Trung Bui, Chen Sun

Primary Area: representation learning for computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: concept learning, concept bottleneck models, vision-language models

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: Concept discovery and learning enable pretrained vision-language models to predict interpretable visual concepts

Abstract: Vision-language models (VLMs) pretrained on web-scale data excel at recognizing complex visual objects. However, it remains mysterious if and how the VLMs learn and utilize rich semantic information of visual concepts, such as colors and shapes, for recognition. While some prior work concluded that pretrained VLMs do not capture interpretable concepts, other work observed that leveraging the concept-based text prompts improves visual recognition accuracy, and appears to offer some degree of interpretability. In this paper, we aim to address this discrepancy and understand pretrained VLMs' true capability of encoding interpretable visual concepts. We identify that the discrepancies on concept definition and concept prompting (class-conditioned or class-agnostic) lead to different observations in prior works, and (class-conditioned) concept prompts that provide discriminative information for visual recognition are often not interpretable. To address these challenges, we propose a new framework to jointly discover and learn interpretable visual concepts from pretrained VLMs. Our discovered concepts are class-agnostic, and selected based on the visual discriminability as measured by mutual information between images and concepts. We then propose a self-supervised framework to efficiently fine-tune a VLM to better recognize the discovered concepts. Through extensive quantitative and human evaluations, we demonstrate that our concept discovery and learning (CDL) framework significantly improves the interpretability of the discovered concepts, while achieving state-of-the-art performance on concept-based visual recognition. All code and data related to this paper will be made public.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5828