A Token-Level Decoding Algorithm For Ensembling Models Across Vocabulary Sizes

ACL ARR 2025 February Submission5442 Authors

Abstract:

Model ensembling is a technique to combine the predicted distributions of two or more models, often leading to improved robustness and performance.
For ensembling in text generation, the next token's probability distribution is derived from a weighted sum of the distributions of each individual model.
This requires the underlying models to share the same subword vocabulary, limiting the applicability of ensembling as many open-sourced models have distinct vocabularies.
In research settings, experimentation or upgrades to vocabularies may introduce multiple vocabulary sizes.
This paper proposes an inference-time only algorithm that allows for ensembling models with different vocabularies, without the need to learn additional parameters or alter the underlying models.
Instead, the algorithm ensures that tokens generated by the ensembled models agree in their surface form.
We apply this technique to combinations of traditional encoder-decoder models and decoder-only LLMs and evaluate on machine translation.
In addition to expanding to model pairs that were previously incapable of token-level ensembling, our algorithm frequently improves translation performance over either model individually.

Paper Type: Long

Research Area: Generation

Research Area Keywords: generation, language modeling, machine translation, nlp applications

Contribution Types: NLP engineering experiment, Publicly available software and/or pre-trained models

Languages Studied: English, German, Spanish, Czech, Ukrainian

Submission Number: 5442