Predictive Task Guidance with Artificial Intelligence in Augmented Reality

Benjamin Rheault, Shivvrat Arya, Akshay Vyas, Jikai Wang, Rohith Peddi, Brett Benda, Vibhav Gogate, Nicholas Ruozzi, Yu Xiang, Eric D. Ragan

Abstract: We put forward an augmented reality (AR) system using artificial intelligence (AI) to guide users through real-world procedures. The system uses the cameras of the headset to recognize objects and their positions in real time. It constructs a model to predict: (i) the task being completed, (ii) the step of the task they are on, and (iii) whether they have made any errors. By updating this model in real-time as the user completes the task, the system automatically updates the instructions and cues provided to the user. This system represents a step towards ubiquitous task guidance via AR.