InfiniSST: Simultaneous Translation of Unbounded Speech with Large Language Model

ACL ARR 2025 February Submission6632 Authors

Abstract: Simultaneous translation of unbounded streaming speech remains a challenging problem due to the need for effectively processing the history speech context and past translations so that quality and latency, including computation overhead, can be balanced. Most prior works assume pre-segmented speech, limiting their real-world applicability. In this paper, we propose InfiniSST, a novel approach that formulates SST as a multi-turn dialogue task, enabling seamless translation of unbounded speech. We construct translation trajectories and robust segments from MuST-C with multilatency augmentation during training and develop a key-value (KV) cache management strategy to facilitate efficient inference. Experiments on MuST-C En-Es, En-De, and En-Zh demonstrate that InfiniSST reduces computation-aware latency by 0.5 to 1 second while maintaining the same translation quality compared to baselines. Ablation studies further validate the contributions of our data construction and cache management strategy.

Paper Type: Long

Research Area: Machine Translation

Research Area Keywords: efficient inference for MT, speech translation

Contribution Types: NLP engineering experiment, Approaches low compute settings-efficiency, Data resources, Data analysis

Languages Studied: English, Chinese, German, Spanish

Submission Number: 6632