A Comparative Study on Vocabulary Reduction for Phrase Table Smoothing

Yunsu Kim, Andreas Guta, Joern Wuebker, Hermann Ney

Abstract: This work systematically analyzes the smoothing effect of vocabulary reduction for phrase translation models. We extensively compare various word-level vocabularies to show that the performance of smoothing is not significantly affected by the choice of vocabulary. This result provides empirical evidence that the standard phrase translation model is extremely sparse. Our experiments also reveal that vocabulary reduction is more effective for smoothing large-scale phrase tables.