Let Your Heart Speak in its Mother Tongue: Multilingual Captioning of Cardiac Signals

Dani Kiyasseh, Tingting Zhu, David A. Clifton

Keywords: multilingual representation learning, cardiac signal captioning

Abstract: Cardiac signals convey a significant amount of information about the health status of a patient. Upon recording these signals, cardiologists are expected to manually generate an accompanying report to share with physicians and patients. Generating these reports, however, can be time-consuming and error-prone, while also exhibiting a high degree of intra- and inter-physician variability. To address this, we design a neural, multilingual, cardiac signal captioning framework. In the process, we propose a discriminative multilingual representation learning method, RTLP, which randomly replaces tokens with those from a different language and tasks a network with identifying the language of all tokens. We show that RTLP performs on par with state-of-the-art pre-training methods such as MLM and MARGE, while generating more clinically accurate reports than MLM. We also show that, with RTLP, multilingual fine-tuning can be preferable to its monolingual counterpart, a phenomenon we refer to as the \textit{blessing of multilinguality}.

One-sentence Summary: A multilingual cardiac signal captioning framework that generates clinical reports in multiple languages when presented with cardiac signals