Throughout Procedural Transformer for Online Action Detection and Anticipation

Haomiao Yuan, Yi Chen, Zheyan Ji, Zhichao Zheng, Yanhui Gu, Junsheng Zhou

Abstract: Recent researches have yielded promising results by integrating online action detection and action anticipation tasks to explore the correlations between past, present and future. However, these approaches treat incomplete historical information equally and neglect intrinsic connections between actions, resulting in a limited perception of the throughout evolution. To address this limitation, we reconsider the patterns and dependencies in event evolution, innovatively constructing a comprehensive deductive process that inscribes the entire temporal spectrum via procedural features. Here, we propose the Throughout Procedural Transformer (TPT) comprising Procedural History Evolution Encoder and Progressive Deduction Decoder, to thoroughly span the entirety of time from history to the future through procedural modeling. TPT utilizes long-term procedural history acquired through procedure sampling to model long-term procedural future, thereby enhancing cognitive inference ability by enriching short-term history and short-term future with a broad grasp of throughout event evolution. We conduct extensive experiments to evaluate TPT on five demanding benchmarks THUMOS’14, TVSeries, FineAction, HACS and EPIC-Kitchens-100 for online action detection and anticipation tasks, demonstrating significant improvements over existing methods.

External IDs:dblp:journals/tcsv/YuanCJZGZ25