# Increment3D

This is the official implementation of the paper **Incremental3D: Incremental 3D Scene Generation with Scene Graph  for Immersive Teleoperation**. It can incrementally generate new objects with accurate layout and shape  at 38 Hz. It is well suited for immersive telerobotcs.


## Setup
### Environment

We have tested it on Ubuntu 20.04 
```javascript
conda create -n Incremental3D python=3.9
conda activate Incremental3D
conda install -y -c conda-forge cudatoolkit=11.7 cudatoolkit-dev=11.7
pip install --index-url https://download.pytorch.org/whl/cu117 torch==2.0.1+cu117 torchvision==0.15.2+cu117
pip install open3d==0.19.0 pyglet==2.1.2 PyOpenGL==3.1.0
pip install pytorch3d==0.7.4
pip install pyquaternion==0.9.9 pyrender==0.1.45 imageio==2.37.0
pip install -r requirements.txt
```

Setup additional Chamfer Distance calculation for evaluation:

```javascript
cd ./extension
python setup.py install
```
### Dataset
1. Download the <a href="https://tianchi.aliyun.com/specials/promotion/alibaba-3d-scene-dataset">3D-FRONT dataset</a> from their official site.

2. Preprocess the dataset following  <a href="https://github.com/nv-tlabs/ATISS#data-preprocessing">ATISS</a>.
3. Download [3D-FUTURE-SDF](https://www.campar.in.tum.de/public_datasets/2023_commonscenes_zhai/3D-FUTURE-SDF.zip). This is processed by ourselves on the 3D-FUTURE meshes using tools in [SDFusion](https://github.com/yccyenchicheng/SDFusion).

4. Follow [this page](./SG-FRONT.md) for downloading SG-FRONT and accessing more information.
5. Download reconstructed meshes and codes by DeepSDF https://www.campar.in.tum.de/public_datasets/2023_commonscenes_zhai/DEEPSDF_reconstruction.zip
6. Create a folder named `FRONT`, and copy all files to it.


The structure should be similar like this:
```
FRONT
|--3D-FRONT
|--3D-FRONT_preprocessed (by ATISS)
|--threed_front.pkl (by ATISS)
|--3D-FRONT-texture
|--3D-FUTURE-model
|--3D-FUTURE-scene
|--3D-FUTURE-SDF
|--DEEPSDF_reconstruction
|--All SG-FRONT files (.json and .txt)
```
### Models
The weight will be released upon paper acceptance.

## Training

To train the models, run:

```
cd scripts
python train_3d_increment_g.py  --exp  /home/xxx/data_weight/increment3dG/all --room_type all --dataset /media/xxx/xxx/FRONT --batchSize 4 --workers 4 --loadmodel False --loadepoch 30 --nepoch 31 --large False --network_type increment3dG
```
`--exp`: where you store the models.

`--room_type`: rooms to train, e.g., livingroom, diningroom, bedroom, and all.

`--batch_size`: the batch size for the layout branch training. 

`--large` : default is `False`, `True` means more concrete categories.

`--network_type`: increment3dG, baseline1, baseline2


## Evaluation

To evaluate the models run:
```
cd scripts
python eval_3d_increment_g.py --exp /home/xx/data_weight/increment3dG/all --room_type all --epoch 30 --visualize True --no_stool False
```
`--exp`: where you store the models.

`--room_type`: rooms to train, e.g., livingroom, diningroom, bedroom, and all.

`--batch_size`: the batch size for the layout branch training. 

`--visualize`: visualize the 3D scene graph and 3D scene. 


## Acknowledgements

This repository is based on <a href="https://github.com/ymxlzgy/commonscenes">CommonScenes</a> and <a href="https://github.com/he-dhamo/graphto3d">Graph-to-3D</a>. We thank the authors for making their code available.
