# OpenDAS: Open-Vocabulary Domain Adaptation for Segmentation

This repository consists of the following structure:<br>
```
    .
    ├── multimodal-prompt-learning        # All multi-modal prompt learning methods
    ├── ov-seg-opendas                    # Integration of OpenDAS into OVSeg pipeline
    ├── scannetpp-openmask3d-opendas      # Integration of OpenDAS into OpenMask3D pipeline for ScanNet++
    ├── analyze_data                      # Jupyter notebooks for analyzing logs and other results
    └── README.md
```

## Highlights

![main figure](./figures/‎teaser_opendas_final.jpeg)
> **<p align="justify"> Abstract:** *Recently, Vision-Language Models (VLMs) have advanced segmentation techniques by shifting from the traditional segmentation of a closed-set of predefined object classes to open-vocabulary segmentation (OVS), allowing users to segment novel classes and concepts unseen during training of the segmentation model. However, this flexibility comes with a trade-off: fully-supervised closed-set methods still outperform OVS methods on base classes, that is on classes on which they have been explicitly trained. This is due to the lack of pixel-aligned training masks for VLMs (which are trained on image-caption pairs), and the absence of domain-specific knowledge, such as autonomous driving. Therefore, we propose the task of open-vocabulary domain adaptation to infuse domain-specific knowledge into VLMs while preserving their open-vocabulary nature. By doing so, we achieve improved performance in base and novel classes. Existing VLM adaptation methods improve performance on base (training) queries, but fail to fully preserve the open-set capabilities of VLMs on novel queries. To address this shortcoming, we combine parameter-efficient prompt tuning with a triplet-loss-based training strategy that uses auxiliary negative queries. Notably, our approach is the only parameter-efficient method that consistently surpasses the original VLM on novel classes. Our adapted VLMs can seamlessly be integrated into existing OVS pipelines, e.g., improving OVSeg by +6.0% mIoU on ADE20K for open-vocabulary 2D segmentation, and OpenMask3D by +4.1% AP on ScanNet++ Offices for open-vocabulary 3D instance segmentation without other changes.* </p>

## Main Contributions
1) **Domain Adaptation for Open-Vocabulary Segmentation:** We introduce a new task, namely "domain adaptation for open vocabulary
segmentation".
2) **Triplet Loss for Better Generalization:** We leverage a triplet loss in domain adaptation for open-
vocabulary segmentation for better generalization.
3) **Simpler Architecture**: We utilize a simpler yet efficient prompt tuning architecture, where we adapt each modality independently.
4) **Application to 2D and 3D open-vocabulary segmentation**: We demonstrate our model's adaptation capabilities in 2D and 3D segmentation settings.




