#### [OPEN TEACH: A Versatile Teleoperation System for Robotic Manipulation]



This is the official implementation of the Open Teach including unity scripts for the VR application, teleoperation pipeline and demonstration collection pipeline.

Open Teach consists of two parts. 

- [x] Teleoperation using Meta Quest 3 and data collection over a range of robot morphologies and simulation environments.

- [x] Policy training for various dexterous manipulation tasks across different robots and simulations.

##### VR Application Installation.

This can be done in two ways.

1. You can install the application directly into your Quest using the APK file we provide using SideQuest.
2. Inorder to have more flexibility and modify the VR APK we have also released the source code corresponding to it. The APK is made in Unity 2021.3.5f1. To ensure clean builds with no errors using this version is recommended.
3. For building the apk from source , add the zipped code files to Unity and within the settings in File menu run Build and Run. The targeted device in Build settings should be the Quest you are connecting to. Also users might need Meta account with Meta Quest Developer Hub and will need to enable Developer mode for the Meta Quest Application. 
4. To setup the VR Application in Oculus Headset, enter the IP Address of the robot server. The robot server and the Oculus Headset should be in the same network. 

##### User Interface

###### Single Robot Arm + Robot Hand: 

​	Since the Robot Hand is a Right Hand, the user controls to switch modes are in Left Hand to have seemless keypoint stream happening from the Right Hand. 

| Pinch ( Left Hand) | Mode                 | Stream Border Color |
| ------------------ | -------------------- | ------------------- |
| Index Pinch        | Only Hand Mode       | Green               |
| Middle Pinch       | Arm + Hand Mode      | Blue                |
| Ring Pinch         | Pause                | Red                 |
| Pinky Pinch        | Resolution Selection | Black               |

######  Multi Robot Arm (Bimanual):

Since both the hands are being used here for teleoperation and gripper mode selection we use pinches in both the hands. Due to the noise in hand pose detection while moving the hands , for better detection of pinches we use keypoint distance threshold based approach between two fingers. 

| Pinch ( On Both Hands ) | Mode                                                         | Stream Border Color |
| ----------------------- | ------------------------------------------------------------ | ------------------- |
| Index Pinch             | Start the Teleop ( Only used at the start of the teleoperation ) | Green               |
| Middle Pinch            | Pause/Resume the Robot                                       | Red                 |
| Ring Pinch              | Pause/Resume the Robot                                       | Red                 |
| Pinky Pinch             | Gripper Open/Close                                           | Yellow              |

##### Server Code Installation 

Install the conda environment from the yaml file in the codebase. 

`conda env create -f environment.yml`

This will install all the dependencies required for the server code.  

After installing all the prerequisites, you can install this pipeline as a package with pip:

`pip install -e . `

You can test if it had installed correctly by running ` import openteach` from the python shell.

##### Robot Controller Specific Information 

1. Franka Controler we use are available at [DeOxys](https://github.com/UT-Austin-RPL/deoxys_control). 
2. For Allegro & Kinova we use open source controllers of [Kinova](https://github.com/NYU-robot-learning/Kinova-Arm-Controller-DIME) and [Allegro](https://github.com/NYU-robot-learning/Allegro-Hand-Controller-DIME)
3. For XArm we use XArmAPI from [xArm-SDK](https://github.com/xArm-Developer/xArm-Python-SDK).
4. The open source implementation for LIBERO env is available [here](https://github.com/Lifelong-Robot-Learning/LIBERO?tab=readme-ov-file#Installation).
5. Allegro-Simulation is a custom simulation env suite created by us and is built on [IsaacGym](https://developer.nvidia.com/isaac-gym). 

##### Installation Specific Information

1. For Allegro Sim, installing the dependencies of isaacgym followed by the dependencies of openteach is recommended.

2. For LIBERO sim , follow the official installation documentation. Then in a python shell,

   `import libero`

   `from libero.libero import benchmark, get_libero_path`

   when prompted for the following line 

   `answer = input("Do you want to specify a custom path for the dataset folder? (Y/N):").lower()`

   input N. 

   Then the environment is ready for teleoperation and data collection.

##### Running the teleop

###### For Robots

###### Without Xela Sensors

To use the OpenTeach teleop module with robots, open the VR Application in your Oculus Headset. On the robot server side, start the controllers first for robots followed by the following command to start the teleop :

`python teleop.py robot=allegro/kinova/franka/allegro_franka/allegro_kinova/bimanual`

###### With Xela Sensors

`python teleop.py robot=xela_hand_franka run_xela=True`

###### For Simulation 

To use the Open-Teach teleop module, open the VR Application in your Oculus Headset. On the server side, use the following command to start the teleop :

`python teleop.py robot=libero_sim/allegro_sim sim_env=True`

##### Data Collection

The Data Collection module saves the robot states , cameras sensors output as a video and sensor states.

###### For Robots 

###### Without Xela Sensors

`python data_collect.py robot=allegro/kinova/franka/allegro_franka/allegro_kinova/bimanual demo_num=1`

###### With Xela Sensors

`python data_collect.py robot=xela_hand_franka demo_num=1 is_xela=True`

###### For Simulation

`python3 data_collect.py robot=allegro_sim/libero_sim demo_num=1 sim_env=True sim=True`

###### Note: Remember to enter your network IP on the server config network.yaml







