Unsupervised Open-Set Task Adaptation Using a Vision-Language Foundation Model

Youngjo Min, Kwangrok Ryoo, Bumsoo Kim, Taesup Kim

Keywords: Unsupervised task adaptation, open-set task adaptation, Vision-language model

TL;DR: We propose a simple yet effective algorithm named UOTA, which significantly improves a vision-language foundation model's classification performance in a real-world, open-set unlabeled data setting without using any source data or model.

Abstract:

Human-labeled data is essential for developing deep learning models that can attain human-level recognition.
However, the cost associated with annotation presents a substantial practical challenge when deploying these models in real-world applications.
Recently, vision-language models like CLIP have shown remarkable zero-shot learning abilities due to vision-language pre-training.
While these models can adapt to various tasks without task-specific human-labeled data, fine-tuning them with such data, though beneficial, can be impractical in cost-sensitive situations.
In this paper, we propose an alternative method that harnesses vast amounts of open-set unlabeled data from the wild to establish a robust image classification model suitable for real-world scenarios.
Our proposed algorithm, Unsupervised Open-Set Task Adaptation (UOTA), offers a straightforward and practical solution, fully capitalizing on the pre-trained CLIP model to enhance its performance by exhaustively utilizing open-set unlabeled data.
We substantiate the effectiveness of our contributions through comprehensive experiments conducted on open-set domain adaptation (OSDA) benchmarks that are relevant to our framework.
Remarkably, without leveraging any source domain model or labeled source data, our method substantially enhances CLIP's classification performance and attains state-of-the-art results on these benchmarks.

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3125