TRITON: Neural Neural Textures for Better Sim2Real

Ryan D Burgert, Jinghuan Shang, Xiang Li, Michael S Ryoo

Keywords: differentiable rendering, sim2real, image translation

TL;DR: We use differentiable rendering and unsupervised image translation to enhance the photorealism of robotic simulations.

Abstract: Unpaired image translation algorithms can be used for sim2real tasks, but many fail to generate temporally consistent results. We present a new approach that combines differentiable rendering with image translation to achieve temporal consistency over indefinite timescales, using surface consistency losses and neu- ral neural textures. We call this algorithm TRITON (Texture Recovering Image Translation Network): an unsupervised, end-to-end, stateless sim2real algorithm that leverages the underlying 3D geometry of input scenes by generating realistic- looking learnable neural textures. By settling on a particular texture for the objects in a scene, we ensure consistency between frames statelessly. TRITON is not lim- ited to camera movements — it can handle the movement and deformation of ob- jects as well, making it useful for downstream tasks such as robotic manipulation. We demonstrate the superiority of our approach both qualitatively and quantita- tively, using robotic experiments and comparisons to ground truth photographs. We show that TRITON generates more useful images than other algorithms do. Please see our project website: tritonpaper.github.io

Student First Author: yes

Website: https://tritonpaper.github.io

Code: https://github.com/TritonPaper/TRITON