# HELIOS: Hierarchical Exploration for Language-grounded Interaction in Open Scenes

## Environment Setup

```bash
mamba create -y -n helios python=3.9
mamba activate helios

# compile dependencies
mamba install -y -c pytorch -c nvidia pytorch=1.13.1 torchvision=0.14.1 torchaudio=0.13.1 pytorch-cuda=11.7 mkl=2024.0.0 fairscale
mamba install -y -c nvidia cuda=11.7 cuda-nvcc=11.7 cuda-cccl=11.7
mamba install -y gxx=8.5 cmake pybind11 python-devtools make git git-lfs

# runtime dependencies
mamba install -y -c pyg pyg
mamba install -y -c aihabitat withbullet habitat-sim=0.2.5 headless
mamba install -y pandas hydra-core gym=0.22.0 trimesh=3 tensorboard ifcfg python-lmdb simplejson threadpoolctl pybullet\
        plotly dash addict scikit-learn scikit-image scikit-fmm pinocchio loguru yacs openai-clip fvcore iopath pycocotools\
        timm transformers==4.26.0 einops spacy pycocoevalcap braceexpand gdown natsort pytorch_geometric yapf nvtx
mamba install -y -c pyg pytorch-cluster 
pip install --no-deps opencv-python==4.11.0.86
pip install --no-deps supervision==0.25.1
pip install --no-deps webdataset===0.1.40 faster-fifo open3d sophuspy===0.0.8 salesforce-lavis decord\
        git+https://github.com/chahyon-ku/frontier_exploration.git \
        git+https://github.com/ChaoningZhang/MobileSAM.git\
        git+https://github.com/naokiyokoyama/depth_camera_filtering
pip install --no-deps git+https://github.com/facebookresearch/pytorch3d.git 

# home robot
git clone git@github.com:facebookresearch/home-robot.git

# download data
export HOME_ROBOT_ROOT=$PWD
./home-robot/download_data.sh -y
git lfs install
cd data/datasets/ovmm
git lfs pull
cd -
cd data/hssd-hab
git lfs pull
cd -
cd data/objects
git lfs pull
cd -
mkdir -p data/checkpoints
cd data/checkpoints
wget --no-check-certificate https://dl.fbaipublicfiles.com/habitat/data/baselines/v1/ovmm_baseline_home_robot_challenge_2023_v0.2.zip
unzip ovmm_baseline_home_robot_challenge_2023_v0.2.zip -d  ovmm_baseline_home_robot_challenge_2023_v0.2
cd -

cd home-robot
export HOME_ROBOT_ROOT=$PWD
pip install --no-deps -e src/home_robot
pip install --no-deps -e src/home_robot_sim
# submodules
git submodule update --init --recursive -f src/home_robot/home_robot/perception/detection/detic/Detic src/third_party/detectron2 src/third_party/contact_graspnet  src/third_party/habitat-lab src/third_party/MiDaS src/home_robot/home_robot/agent/imagenav_agent/SuperGluePretrainedNetwork src/home_robot/home_robot/perception/detection/grounded_sam/Grounded-Segment-Anything/
## habitat-lab
pip install --no-deps -e src/third_party/habitat-lab/habitat-lab
pip install --no-deps -e src/third_party/habitat-lab/habitat-baselines
## detectron2
pip install --no-deps -e src/third_party/detectron2
# If this doesn't work you may need to install a different version of setup tools: pip install --upgrade setuptools==75.8.2
## grounded-sam
pip install --no-deps -e src/home_robot/home_robot/perception/detection/grounded_sam/Grounded-Segment-Anything/segment_anything
pip install --no-deps -e src/home_robot/home_robot/perception/detection/grounded_sam/Grounded-Segment-Anything/GroundingDINO
wget https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth -O src/home_robot/home_robot/perception/detection/grounded_sam/Grounded-Segment-Anything/GroundingDINO/groundingdino_swint_ogc.pth
## detic
mkdir src/home_robot/home_robot/perception/detection/detic/Detic/models/
wget --no-check-certificate https://dl.fbaipublicfiles.com/detic/Detic_LCOCOI21k_CLIP_SwinB_896b32_4x_ft4x_max-size.pth -O src/home_robot/home_robot/perception/detection/detic/Detic/models/Detic_LCOCOI21k_CLIP_SwinB_896b32_4x_ft4x_max-size.pth
## MiDaS
wget https://github.com/isl-org/MiDaS/releases/download/v3_1/dpt_beit_large_512.pt -O src/third_party/MiDaS/weights/dpt_beit_large_512.pt
cd -

# cd third_party
# git clone git@github.com:WongKinYiu/yolov7.git
# cd ..

pip install git+https://github.com/nerfstudio-project/gsplat.git

# ours
pip install --no-deps -e third_party/vlfm
mkdir weights -p
wget https://github.com/WongKinYiu/yolov7/releases/download/v0.1/yolov7-e6e.pt -O weights/yolov7-e6e.pt
wget https://github.com/ChaoningZhang/MobileSAM/raw/master/weights/mobile_sam.pt -O weights/mobile_sam.pt
wget https://github.com/bdaiinstitute/vlfm/raw/main/data/pointnav_weights.pth -O weights/pointnav_weights.pt
pip install --no-deps -e .

mkdir home-robot/configs
cp configs/example_cat_map.json home-robot/configs/example_cat_map.json
```

## Experiments
Scripts to run full experiments are in python `scripts/experiments/`. After completing experiments use `scripts/fuse_metrics.py` and `scripts/fuse_metrics_n_picks.py` to generate the results for each setting across all the episodes with standard error. `scripts/fuse_metrics_n_picks.py` is currently set-up to generate results for 1 pick, change `n_picks` from 1 to 5 and (optionally) rename the save locations to generate the fused results for 5 picks.

For running different configurations use `python scripts/evaluate.py` with overrides from `configs/evaluate.yaml`, e.g.

```bash
### helios
python scripts/evaluate.py agent/gaze=hierarchical_gaze catch_exceptions=True num_episodes=1 visualize_3dgs=True

### w/o global objective
python scripts/evaluate.py agent/gaze=hierarchical_gaze catch_exceptions=True num_episodes=1 visualize_3dgs=True use_global_objective=False

### Trusting agent
python scripts/evaluate.py agent/gaze=integrated_simple_gaze catch_exceptions=True num_episodes=1 

### helios with gt semantics
python scripts/evaluate.py agent/gaze=hierarchical_gaze catch_exceptions=True num_episodes=1 visualize_3dgs=True gt_semantics=True
```

Note that running with visualizations is significantly slower than without.