Stream-level Latency Evaluation for Simultaneous Machine Translation

Jorge Civera, Alfons Juan-Císcar, Javier Iranzo-Sánchez

Abstract:

Simultaneous machine translation has recently
gained traction thanks to significant quality im-
provements and the advent of streaming ap-
plications. Simultaneous translation systems
need to find a trade-off between translation
quality and response time, and with this pur-
pose multiple latency measures have been pro-
posed. However, latency evaluations for si-
multaneous translation are estimated at the sen-
tence level, not taking into account the sequen-
tial nature of a streaming scenario. Indeed,
these sentence-level latency measures are not
well suited for continuous stream translation,
resulting in figures that are not coherent with
the simultaneous translation policy of the sys-
tem being assessed. This work proposes a
stream-level adaptation of the current latency
measures based on a re-segmentation approach
applied to the output translation, that is suc-
cessfully evaluated on streaming conditions
for a reference IWSLT task.