Action Prediction Network with Auxiliary Observation Ratio Regression

Cuiwei Liu, Yiming Gao, Zhaokui Li, Chong Du, Fang Liu, Xiangbin Shi

Abstract: This paper focuses on predicting the category of an ongoing action with incomplete observations. We propose a novel Action Prediction Network with Auxiliary Observation Ratio regression (AORAP Net), which enhances the discriminative power of partial video features by encoding the prior knowledge of complete actions. The proposed AORAP Net consists of an encoder, an observation ratio regression module, and an action classifier. The encoder transfers global action information from full videos to construct enhanced partial video features. The observation ratio regression module is developed to achieve an auxiliary task of inferring the progress level of an ongoing action. We demonstrate that this module can guide the encoder to learn enhanced features similar to full video features and discriminative for action prediction. The classifier makes the final decision of the action category in terms of the enhanced features. Considering the fact that there is not enough discriminative information at the early stage of actions, a new classification loss is designed to adapt to the action prediction task and alleviate the over-fitting of training videos. Extensive experiments on the BIT-Interaction dataset and the UT-Interaction dataset validate the effectiveness of the proposed method.