## ❄️ PerSense
Implementation of the paper **"PerSense: Personalized Instance Segmentation in Dense Images"** 🚩

## 🛠️ Requirements

### Installation
Similar to SAM and PerSAM, our code requires pytorch>=1.7 and torchvision>=0.8. For compatibility check [here](https://pytorch.org/get-started/locally/).
unzip the folder and Create conda environment following the instructions given below:

    cd PerSense

    conda create -n persense python=3.8
    conda activate persense

    conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia 
    (Change it as per your platform following the link given above)

    pip install -r requirements.txt

### Dataset
Please download our **PerSense-D dataset** from [here](https://drive.google.com/file/d/1FfY82qZt6h8sEEbko7T-gOn_zHrMPMOl/view?usp=sharing). Unzip the dataset and organize it as follows:

    data/
    |-- Images/

### 🔩 Model Weights
Please download pretrained weights of SAM from [here](https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth).
Also, download weights for DSALVANet pretrained on FSC-147 from [here](https://drive.google.com/file/d/1oIuG_rKtadicnQ8y2ibSTjnTzlnW0kSP/view?usp=sharing) and weights for GroungdingDINO from [here](https://drive.google.com/file/d/1klsbCpfhuZvdVGQ7aAfr_2aL8eSSfhVs/view?usp=sharing). Download weights for CounTR pretrained on FSC-147 dataset from [here](https://drive.google.com/file/d/1CzYyiYqLshMdqJ9ZPFJyIzXBa7uFUIYZ/view?pli=1). For ViPLLaVA (VLM) weights are automatically fetched through the code.

    data/
    sam_vit_h_4b8939.pth

    DSALVANet/checkpoints/
    checkpoint_200.pth

    GroundingDINO/weights/
    groundingdino_swint_ogc.pth

    CounTR/output_allnew_dir/
    FSC147.pth

## 🏃‍♂️ Getting Started

#### PerSense
To evaluate PerSense on DSALVANet, just run the following command: 

    python persense.py (add argument --visualize True for visualizing masks overlaid on original image)

To evaluate PerSense on CounTR, just run the following command: 

    python persense.py --fsoc countr (add argument --visualize True for visualizing masks overlaid on original image)

#### Grounded-SAM
To evaluate Grounded-SAM in our scenario, just run the following command:

    python groundedsam.py (add argument --visualize True for visualizing masks overlaid on original image)

#### Matcher
Follow installation instructions given at https://github.com/aim-uofa/Matcher. Organize PerSense-D as Matcher/data/Images/. Place matcher_test.py and eval_miou.py under Matcher/. Run the following command to test Matcher on PerSense-D dataset:

    python matcher_test.py

#### PerSAM
persense conda environment can be used for PerSAM. Just clone the PerSAM repo available at https://github.com/ZrrSkywalker/Personalize-SAM. Place PerSense-D in the dataset directory and run following:

    persam_f_multi_obj.py

#### Evaluate mIoU
To evaluate mIoU, just run the following command:

    python eval_miou.py --pred_path PerSense or groundedsam or Matcher or persam

## License
[![Code License](https://img.shields.io/badge/Code%20License-Apache_2.0-green.svg)](https://github.com/tatsu-lab/stanford_alpaca/blob/main/LICENSE)
[![Data License](https://img.shields.io/badge/Data%20License-CC%20By%20NC%204.0-red.svg)](https://github.com/tatsu-lab/stanford_alpaca/blob/main/DATA_LICENSE)  

**Usage and License Notices**: The data, model and pretrained checkpoints are intended and licensed for research use only. They are also restricted to uses that follow the license agreement of LLaMA, Vicuna and GPT-4. The dataset is CC BY NC 4.0 (allowing only non-commercial use) and models trained using the dataset should not be used outside of research purposes.



