# MambaTrack



### Environment Settings 
* **Install environment using conda**
```
conda create -n mamba_fetrack python=3.10.13
conda activate mamba_fetrack
```


 * **Install the package for Vim**
```
conda install cudatoolkit==11.8 -c nvidia
pip install torch==2.1.1 torchvision==0.16.1 torchaudio==2.1.1 --index-url https://download.pytorch.org/whl/cu118
conda install -c "nvidia/label/cuda-11.8.0" cuda-nvcc
conda install packaging
pip install -r vim_requirements.txt
```
* **Install the mamba-1.1.1 and casual-conv1d-1.1.3 for mamba**

Download the [mamba-1.1.1](https://github.com/state-spaces/mamba/releases/download/v1.1.1/mamba_ssm-1.1.1+cu118torch2.1cxx11abiFALSE-cp310-cp310-linux_x86_64.whl) and [source code](https://github.com/state-spaces/mamba/archive/refs/tags/v1.1.1.zip) and place it in the project path of Mamba_FETrack. Go to source code and install the corresponding environment.
```
cd mamba-1.1.1
pip install .
```

Download the [casual-conv1d-1.1.3](https://github.com/Dao-AILab/causal-conv1d/releases/download/v1.1.3/causal_conv1d-1.1.3+cu118torch2.1cxx11abiFALSE-cp310-cp310-linux_x86_64.whl) and [source code](https://github.com/Dao-AILab/causal-conv1d/archive/refs/tags/v1.1.3.zip) and place it in the project path of Mamba_FETrack.  Go to source code and install the corresponding environment.
```
cd ..
cd causal-conv1d-1.1.3
pip install .
```
    
* **Install the package for tracking**
```
bash install.sh
```

* **Run the following command to set paths for this project**
```
python tracking/create_default_local_file.py --workspace_dir . --data_dir ./data --save_dir ./output
```

* **After running this command, you can also modify paths by editing these two files**
```
lib/train/admin/local.py  # paths about training
lib/test/evaluation/local.py  # paths about testing
```

### Dataset Prepair  
* Download tracking datasets [FELT](https://pan.baidu.com/s/12ur7n1wSDvIWajPQJMd8Kg?pwd=AHUT ) OR FE108[train(Passcode：bhhi)](https://pan.baidu.com/s/1a1lMErDO5UTk5ZOgbZk0bg(bhhi))[test(Passcode：0lsf)](https://pan.baidu.com/s/1ijlrCRQVQKKmpuSgUVOewQ(0lsf)), and put it in `./data`.
* Perform some preprocessing on the dataset.
```
# FE108(for dvs/aps/voxel(RGB/Event images)、 density)
python scripts/event_to_frame_FE108.py
python scripts/density_prepair_fe108.py

# FELT(for density)
python scripts/density_prepair_felt.py
```

### Download Checkpoint  
  Download [pre-trained](https://pan.baidu.com/s/1-5q4hK2LWj16K6R2PHSdPw?pwd=AHUT) and put it under `$/lib/models/mamba_fetrack/pretrained_models`.


### Trained Weights for Model 
Download the trained model weights from [Mamba_FETrack_ep0050.pth](https://pan.baidu.com/s/1rPlPXWgJX4vCA7FqVgDBmQ?pwd=dlut) and put it under `$/output/checkpoints/train/mamba_fetrack/mamba_fetrack_felt` for test directly.



### Training and Testing Script 
```
# train
python tracking/train.py --script mamba_fetrack --config mamba_fetrack_felt --save_dir ./output --mode single --nproc_per_node 1 --use_wandb 0
python tracking/train.py --script mamba_fetrack --config mamba_fetrack_fe108 --save_dir ./output --mode single --nproc_per_node 1 --use_wandb 0

# test
python tracking/test.py mamba_fetrack mamba_fetrack_felt --dataset felt --threads 1 --num_gpus 1
python tracking/test.py mamba_fetrack mamba_fetrack_fe108 --dataset fe108 --threads 1 --num_gpus 1
```



### Evaluation Toolkit 
 * **Evaluation methods on the FELT dataset**
1. open the FELT_eval_toolkit with Matlab (over Matlab R2020).
2. add your [tracking results](https://pan.baidu.com/s/1rPlPXWgJX4vCA7FqVgDBmQ?pwd=dlut) in `$/felt_tracking_results/` and modify the name in `$/utils/config_tracker.m`
3. run `Evaluate_FELT_benchmark_SP_PR_only.m` for the overall performance evaluation, including AUC, PR, NPR.

 * **Evaluation methods on the FE108 dataset**
1. add your [tracking results](https://pan.baidu.com/s/1rPlPXWgJX4vCA7FqVgDBmQ?pwd=dlut) in `$/output/test/tracking_results/mamba_fetrack/`
2. run `$/tracking/analysis_results.py` for the overall performance evaluation, including AUC, PR, NPR.


### Notice:
The code of the main idea in our paper is as follows:
1.The special event branch(conclude DSSM module and event-adaptive state transition mechanism) is located in  `$/lib/models/mamba_fetrack/models_mamba.py`:
```
def forward_features_event(self, z, x, z_density, x_density, inference_params=None, if_random_cls_token_position=False, if_random_token_rank=False):
```
2.The Gated Projection Fusion(GPF) module is located in `$/lib/models/mamba_fetrack/mamba_fetrack.py`:
```
def feature_fusion(self, event_feature: torch.Tensor, rgb_feature: torch.Tensor, rho_t: torch.Tensor) -> torch.Tensor:
```
