Redefining Machine Simultaneous Interpretation: From Incremental Translation to Human-like Strategies

Qianen Zhang, Satoshi Nakamura

Keywords: Simultaneous machine translation, Simultaneous interpretation, Machine translation, Large Language Model, Speech processing, LLM fine-tuning, In-context learning, Machine learning

TL;DR: We propose an LLM-based simultaneous MT framework that extends the action space with human-like strategies (cut, drop, summarize, pronominalize) and a latency-aware TTS pipeline, achieving better quality–latency trade-offs than multiple baselines.

Abstract: Simultaneous Machine Translation (SiMT) requires high-quality translations under strict real-time constraints, which traditional encoder-decoder policies with only READ/WRITE actions cannot fully address. We extend the action space of SiMT with four adaptive actions: SENTENCE_CUT, DROP, PARTIAL_SUMMARIZATION and PRONOMINALIZATION, which enable real-time restructuring, omission, and simplification while preserving semantic fidelity. We implement these actions in a decoder-only large language model (LLM) framework and construct training references through action-aware prompting. To evaluate both quality and latency, we further develop a latency-aware TTS pipeline that maps textual outputs to speech with realistic timing. Experiments on the ACL60/60 English-Chinese and English-German benchmarks show that our framework consistently improves semantic metrics (e.g., COMET-da and COMET-KIWI) and achieves lower delay (measured by Average Lagging) compared to reference translations and salami-based baselines. Notably, combining DROP and SENTENCE_CUT yields the best overall balance between fluency and latency. These results demonstrate that enriching the action space of LLM-based SiMT provides a promising direction for bridging the gap between human and machine interpretation.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 10745