Maestro: The Analysis-Simulation Integrated Framework for Mixed Reality

Jingyu Lee, Hyunsoo Kim, Minjae Kim, Byung-Gon Chun, Youngki Lee

Abstract: Mixed reality devices with near-eye displays unlock new possibilities for innovation and user experiences. Mixed reality applications require a new unified framework that enables seamless analysis of the real world and simulation of realistic virtual content. Designing such a framework faces various challenges, including huge programming efforts of analysis and simulation pipelines, and inconsistencies between real-world and virtual content caused by end-to-end processes across pipelines.This paper proposes Maestro, the analysis-simulation integrated framework for mixed reality applications. Maestro provides a programming model for effective application representation and control, aiding runtime optimization. Maestro runtime takes an object-level execution approach to minimize misalignment, integrating both simulation and analysis pipelines for applications to process individual objects based on their latency sensitivity. Our evaluation shows that Maestro improves streaming accuracy by up to 1.6× compared to existing frameworks and effectively expresses nine qualitatively distinct workloads expected in prospective mixed-reality applications.