Learning Procedural Abstractions and Evaluating Discrete Latent Temporal Structure

Karan Goel, Emma Brunskill

Abstract: Clustering methods and latent variable models are often used as tools for pattern mining and discovery of latent structure in time-series data. In this work, we consider the problem of learning procedural abstractions from possibly high-dimensional observational sequences, such as video demonstrations. Given a dataset of time-series, the goal is to identify the latent sequence of steps common to them and label each time-series with the temporal extent of these procedural steps. We introduce a hierarchical Bayesian model called Prism that models the realization of a common procedure across multiple time-series, and can recover procedural abstractions with supervision. We also bring to light two characteristics ignored by traditional evaluation criteria when evaluating latent temporal labelings (temporal clusterings) -- segment structure, and repeated structure -- and develop new metrics tailored to their evaluation. We demonstrate that our metrics improve interpretability and ease of analysis for evaluation on benchmark time-series datasets. Results on benchmark and video datasets indicate that Prism outperforms standard sequence models as well as state-of-the-art techniques in identifying procedural abstractions.

Keywords: learning procedural abstractions, latent variable modeling, evaluation criteria

Code: [ StanfordAI4HI/ICLR2019_evaluating_discrete_temporal_structure](https://github.com/StanfordAI4HI/ICLR2019_evaluating_discrete_temporal_structure)

Data: [Breakfast](https://paperswithcode.com/dataset/breakfast), [JIGSAWS](https://paperswithcode.com/dataset/jigsaws)