Spatiotemporal-Memory-Guided Machine Perception for Augmented Reality

Jianzhe Lin, Shaoyu Chen, Bea Steers, Huy T. Vo, Cláudio T. Silva, Qi Sun

Abstract: Augmented Reality (AR) devices are equipped with advanced sensors such as panoramic cameras. They can see or hear what humans cannot. The superior sensing capabilities - together with machine perception models - enable promising AR-based human assistants in daily tasks. Human task performance progressively enhances with accumulated memory, such as with observed objects' location in working memory, and training in long-term memory. We transform this cognitive nature into real-time machine perception models tailored for human-assisting AR applications. Instead of running machine perception models on each consecutive frame, we leverage the egocentric AR cameras to reconstruct a “4D memory map”, which mimics the human spatiotemporal memory.