A Hybrid Transformer Network for Detection of Risk Situations on Multimodal Life-Log Health Data

Rupayan Mallick, Jenny Benois-Pineau, Akka Zemmari, Marion Pech, Thinhinane Yebda, Hélène Amieva, Laura Middleton

Abstract: The paper is focused on the development of hybrid transformer architectures for the detection of risk events on multimodal data recorded on a person with visual and signal sensors. The proposed two-stream architecture consists of a visual transformer and linear transformer of time series. The linear transformer is benchmarked on the publicly available dataset UCI-HAR. The experiments with our architecture have been conducted on the in-the-wild dataset BIRDS. The hybrid transformer architecture has better empirical performance than the 3D CNNs and RNNs in previous work. The accuracy of detection of risk situations shows an improvement of 10% over the single-stream transformers.