Divergence-based fine pruning of phrase-based statistical translation model

Kangil Kim, Eun-Jin Park, Jong-Hun Shin, Oh-Woog Kwon, Young Kil Kim

Abstract: Highlights•Entropy-based pruning has a limit in selecting a fine distribution of phrase pairs to be pruned in a threshold.•Changing the distribution through other divergence metrics improves pruning efficiency in our preliminary empirical analysis.•Derived problematic factors are fixed divergence distribution and missing impact of word-coupling strength.•We propose a fine pruning method using two parameters to control the factors and analyze their effects to divergence change.•It improves pruning efficiency compared with Entropy-based pruning in practical translations of English, Spanish, and French.