Why Not Use Your Textbook? Knowledge-Enhanced Procedure Planning of Instructional Videos

Kumaranage Ravindu Yasas Nagasinghe, Honglu Zhou, Malitha Gunawardhana, Martin Renqiang Min, Daniel Harari, Muhammad Haris Khan

Abstract: In this paper, we explore the capability of an agent to construct a logical sequence of action steps, thereby as-sembling a strategic procedural plan. This plan is cru-cial for navigating from an initial visual observation to a target visual outcome, as depicted in real-life instructional videos. Existing works have attained partial suc-cess by extensively leveraging various sources of information available in the datasets, such as heavy intermediate visual observations, procedural names, or natural language step-by-step instructions, for features or supervision signals. However, the task remains formidable due to the implicit causal constraints in the sequencing of steps and the variability inherent in multiple feasible plans. To tackle these intricacies that previous efforts have over-looked, we propose to enhance the agent's capabilities by infusing it with procedural knowledge. This knowledge, sourced from training procedure plans and structured as a directed weighted graph, equips the agent to better nav-igate the complexities of step sequencing and its poten-tial variations. We coin our approach KEPP, a novel Knowledge-Enhanced Procedure Planning system, which harnesses a probabilistic procedural knowledge graph ex-tracted from training data, effectively acting as a compre-hensive textbook for the training domain. Experimental evaluations across three widely-used datasets under set-tings of varying complexity reveal that KEPP attains su-perior, state-of-the-art results while requiring only mini-mal supervision. Code and trained model are available at https://github.com/Ravindu-Yasas-Nagasinghe/KEPP