Expert Sample Consensus Applied To Camera Localization for AR Sports Spectators

Shazia Gul, Lewis Baker, Ryan Boult, Steven Mills, Stefanie Zollmann

Abstract: Augmented Reality (AR) applications require robust and accurate localization and tracking of the user or the user’s device. This is important to allow for seamless integration of 3D digital content into the real 3D environment. Robust localization is still a challenge in large-scale dynamic environments such as large sports venues. The purpose of our research is to investigate localization methods that are suitable for dynamic large-scale environments. For this purpose, we explored and evaluated the performance of state-of-the-art methods such as the 6D Camera Localization via 3D Surface Regression (DSAC++) and Expert Sample Consensus (ESAC) method.To investigate the feasibility of these methods, we trained DSAC++ and ESAC using a large-scale stadium image dataset captured when the stadium was empty and accessible for capture. We then used the trained systems for analyzing their robustness and accuracy for a set of different camera sequences in a stadium environment with different levels of crowdedness. Through our experiments, we found that both DSAC++ and ESAC produce acceptable results in an empty stadium. However, the experiments show that the DSAC++ localization robustness decreases for the semi-crowded dataset (75%) and completely struggles (0%) to localize the camera when the environment has a lot of dynamic elements such as a crowd. Our experiments show that ESAC trained on an empty stadium with four experts performs already better on the crowded dataset (localization success rate 87.5%) and further improves when trained with ten experts (93.75%). Our results indicate that ESAC trained on an empty environment can be used for localization in a large-scale dynamic environment.