Deep Learning Anytime Prediction via Enforcing Runtime Monotonicity for Early-Exit Activity Recognition

Xin Liu, Lei Zhang, Wenbo Huang, Di Xiong, Zenan Fu, Hao Wu, Aiguo Song

Abstract: During the past decade, deep neural networks (DNNs) have been widely deployed to a variety of mobile devices for sensor-based human activity recognition (HAR), where their computational budgets are usually dynamic and restricted. Anytime algorithms are especially suitable for such dynamic HAR computing environments, which can return a prediction at an arbitrary time during computation, whose predictive quality gradually evolves as a function of computation time. Early-exit DNNs have recently gained a lot of attention in the context of HAR, which can make intermediate predictions across various exits throughout the whole network. However, most existing early-exit HAR works do not work well under any time setting, since their predictive quality with runtime monotonicity per exit could not be guaranteed as computation time evolves. To mitigate this issue, this article proposes a lightweight post hoc transformation using a formulation of product-of-experts (PoEs), which aims to endow deep models with runtime monotonicity, hence paving an essential step toward truly anytime predictive modeling for early-exit activity recognition. The comprehensive experiments and ablation studies on three publicly available HAR benchmark datasets verify that our anytime predictor can ensure such runtime monotonicity behavior while preserving competitive accuracy on average, with minimal implementation overhead. To the best of the authors’ knowledge, this article is the first to impose runtime monotonicity on predictive quality for early-exit activity recognition. An actual evaluation is performed on a mobile device.