# CoM-V2I

The official codebase for the paper "CoM-V2I: Communication-Efficient Multimodal Cooperative Perception via Codebook Pruning and Multiscale Fusion". 
![CoM-V2I](comv2i.jpg)
- Support both simulation and real-world cooperative perception dataset
    - [x] [V2X-Real](https://mobility-lab.seas.ucla.edu/v2x-real/) (object detction)
    - [x] [V2X-Sim](https://ai4ce.github.io/V2X-Sim/) (map segmentation)

## Overview
- [Data Download](#data-download)
- [Environment Setup](#environment-setup)
- [Dataset Preprocess](#dataset-preprocess)
- [Running instructions](#running-instructions)

## Data Download
### V2X-Real
The V2X-Real dataset is available on [official page](https://mobility-lab.seas.ucla.edu/v2x-real/), which follows with the OPV2V format.
After downloading the data, please put the data in the following structure:

```
├── v2xreal
│   ├── train
|       |── 2023-03-17-15-53-02_1_0
|           |── -1
|               |── 000xxx_camerax.jpeg
|               |── 000xxx.bin
|               |── 000xxx.yaml
│   ├── validate
│   ├── test
```
### V2X-Sim
The V2X-Sim dataset is available on [official page](https://ai4ce.github.io/V2X-Sim/), which follows with the Nuscenes format, which should be reformatted follow as OPV2V format.

The V2X-Sim dataset is only supported by nuscenes_devkit-1.0.9, while package conflicts will occur if installed that version for the later setup. Here, we should create an individual environment to format V2X-Sim dataset as OPV2V format.
- Create conda environment
```shell
conda create -n v2xsim python=3.7
conda activate v2xsim
```
- Install dependencies
```shell
pip install -r opencood/dataset_preprocess/V2XSim/requirements.txt
```
- Reformat dataset
```python
python opencood/dataset_preprocess/V2XSim/create_decent_data.py --dataroot ${V2XSim_FOLDER} --save_dir ${V2XSim_SaveDir}
```
The reformatted dataset should follow the structure shown as below:
```
├── v2xsim-reformat
│   ├── train
|       |── scene1
|           |── 0 
|               |── 000xxx_camerax.npy
|               |── 000xxx.bin
|               |── 000xxx.pkl
|            ...
|           |── 5
│        ...
|       |── scene80
│   ├── test
|       |── scene81
│        ...
|       |── scene100
```
- You can remove it after reformatting V2XSim
```shell
conda remove -n v2xsim --all
```
## Environment setup
Please refer to the following steps for the environment setup:
- Create conda environment
```shell
conda create -n comv2i python=3.9
conda activate comv2i
```
- pytorch installation (cuda version 12.8), please check [website](https://pytorch.org/get-started/previous-versions/) for install other torch version 
```shell
pip3 install torch torchvision
```
- Install other dependencies
```shell
pip install -r requirements.txt
```
- Build up toolkits
```shell
python setup.py develop
python opencood/utils/setup.py build_ext --inplace
```

## Dataset Preprocess
#### V2XReal
Key changes compared to the original version
- The **yaml files** are converted to **pkl files** for accelerating I/O speed.
- Correct the mismatch between camera file name and annotation camera parameters.
- Convert the '.jpeg' camera file to '.npy' file to accelerate data preprocessing.
- Calculate the agent distance matrix for each timestamp. 

Please run the following command before training: 
```python
python opencood/dataset_preprocess/V2XReal/correct_params.py --dataroot ${V2XReal_FOLDER}
```

## Running instructions
For training, please run:
```python
python opencood/tools/train.py --hypes_yaml hypes_yaml/xxx/xxxx.yaml --half --log_tool tensorboard
```

Before inference, please run the codebook pruning command: 
```python
python opencood/tools/codebook_pruning.py --model_dir opencood/logs/xxxx_2025_xx_xx --target_size 16,16,16
```

For inference on detection and segmentation tasks, please run the following commands: 
```python
python opencood/tools/inference_bev.py --model_dir ${CHECKPOINT_FOLDER} --save_vis True
python opencood/tools/inference_seg.py --model_dir ${CHECKPOINT_FOLDER} --save_vis True
```
