# SinCro
This is the PyTorch implementation for the paper SinCro. SinCro is a novel RL framework that leverages 3D-aware representations from single-view RGB input, without requiring camera calibration information or synchronized multi-view images for downstream RL. In this supplementary file, we provide the code for the pre-training of SinCro. Once you train the 3D scene encoder using this file, you can bind off-the-shelf RL algorithms such as DrM for deploying downstream tasks. For more details about the implementation, please refer to our paper and appendix.


## Instructions

Download our code. Then,
```
cd SinCro
conda create -n SinCro python=3.8
conda activate SinCro
pip install -r requirement.txt
```


## Download Dataset
We provide a dataset for the peg-insert-side environment. You can download the dataset from the [drive](https://drive.google.com/file/d/15Fz6JGdS_nyay6tCwQS9osQxt6N2ZkH0/view?usp=sharing), and unzip it to your dataset path. After that, you should create a file "configs/peg_dataset_path.txt" (refer to "configs/peg_dataset_path_template.txt"). In addition to RGB observations, the dataset also includes the output of [Grounded SAM](https://arxiv.org/abs/2401.14159) for object-focused ray sampling.


## Pre-Train 3D Scene Encoder of SinCro Using Multi-View Dataset
For pre-training of the 3D Scene Encoder in the peg-insert-side environment,
```
conda activate SinCro
python MV_run_nerf.py --config configs/peg.txt
```


## Visualization
To visualize the reconstruction results of the trained model, you can use 'MV_visualize.py'.
This file will provide rendered videos and images from all six viewpoints of an episode, as well as quantitative results.
Please modify the variables 'ckpt_folder_dir', 'single_view_input', 'input_view_index', and 'ref_view_index' in this file. Then,
```
conda activate SinCro
python MV_visualize.py
```


## Reference
Our code is based on and modified from the official implementations of [NeRF](https://github.com/yenchenlin/nerf-pytorch) and [CroCo](https://github.com/naver/croco).