RETHINKING POSITIONAL EMBEDDING: A CASE STUDY IN TEMPORAL EVENT SEQUENCE MODELLING

Effat Farhana

Keywords: Sequential Modelliing

TL;DR: An empirical comparison of sinusoidal positional embedding and time-based embedding in transformer. Certain application domains, such as in health and education sector, time-based attention can be

Abstract: In this paper, we present a time-decaying encoding as an alternative to sinusoidal positional encoding in the transformer architecture. We evaluate our approach in the context of an educational domain involving 14,043 question-solving interactions from 1,260 students. We argue that including time-based attention can be beneficial for event sequence modeling applications where the inter-event time intervals and the interpretation of the model's prediction both are crucial.