Flying in XR: Bridging Desktop Applications in eXtended Reality through Deep Learning

Lorenzo Stacchio, Giuseppe Di Maria, Gustavo Marfia

Abstract: The expanding realm of eXtended Reality (XR) has witnessed a surge in 3D experiences across diverse domains, undergoing significant transformations to define novel experiences. However, such experiences are often built from scratch, as there is a lack of tools that support augmenting existing non-immersive interfaces, like Desktop games and simulators, directly into XR. Such shortage is particularly exacerbated in the case of Mixed Reality (MR). Motivated by this, we present a novel middleware, Flying In XR (FIXR), leveraging Deep Learning to visualize and interact with desktop application views into XR. To demonstrate the flexibility of such an approach, we applied FIXR to a commercial Desktop flight simulator, sup-porting an MR experience. It is worth noticing that FIXR could be adapted to communicate with any desktop software with a camera that moves along the depth axis, opening new paths to enable user experiences in XR for a wide spectrum of applications.