UniTorch - Integrating Neural Rendering into Unity

Laura Fink, Daniel Keitel, Marc Stamminger, Joachim Keinert

Abstract: Neural rendering techniques have gained significant attention in recent years for their ability to generate highly realistic and immersive visual content. This paper discusses the current state of game engines regarding their ability to integrate neural modules within their pipelines. We exemplarily chose the popular game engine Unity and the deep learning library LibTorch. As we found a severe gap between commonly used auto-diff, deployment and rendering frameworks regarding interoperability and performance, we designed UniTorch, a plug-in that allows native access from Unity to Torch. We explore the practical integration of neural rendering methods by faithfully reimplementing and extending state-of-the-art methods. We provide detailed implementation guidelines and use it as means to reveal the mentioned gaps through extensive benchmarking experiments.