Knowledge enhanced zero-resource machine translation using image-pivoting

Ping Huang, Jing Zhao, Shilinag Sun, Yichu Lin

Abstract: Zero resource machine translation usually means that there are no parallel corpora in the training of machine translation models, which can be solved with the help of extra information such as images. However, the ambiguity in the text, together with the irrelevant information in images, may cause the problem of translation errors of some key words. In order to alleviate the problem of image-text alignment deviation caused by word ambiguity, we introduce knowledge entities as an extra modality for the source language to enhance the representations of the source text to clarify its semantics. Specifically, we use additional multi-modal information including images and knowledge entities as an auxiliary hint for the source text in the Transformer-based zero-resource translation framework. We also solve the problem of the structural difference between the training and inference stages to handle the cases where there is no longer visual information in the inference stage. The proposed method achieves state-of-the-art BLEU scores in the field of zero-resource machine translation with the image as the pivot.