Time-Aware Circulant Matrices for Question-Based Temporal Localization

Pierfrancesco Bruni, Alex Falcon, Petia Radeva

Abstract: Episodic memory involves the ability to recall specific events, experiences, and locations from one’s past. Humans use this ability to understand the context and significance of past events, while also being able to plan for future endeavors. Unfortunately, episodic memory can decline with age and certain neurological conditions. By using machine learning and computer vision techniques, it could be possible to “observe” the daily routines of elderly individuals from their point of view and provide customized healthcare and support. For example, it could help an elderly person remember whether they have taken their daily medication or not. Therefore, considering the important impact on healthcare and societal assistance, this problem has been recently discussed in the research community, naming it Episodic Memory via Natural Language Queries. Recent approaches to this problem mostly rely on the literature related to similar fields, but contextual information from past and future clips is often unexplored. To address this limitation, in this paper we propose the Time-aware Circulant Matrices technique, which aims at introducing awareness of the surrounding clips into the model. In the experimental results, we present the robustness of our method by ablating its components, and confirm its effectiveness on the Ego4D public dataset, achieving an absolute improvement of more than 1% on R@5.