dMel: Speech Tokenization Made Simple

Richard He Bai, Tatiana Likhomanenko, Ruixiang ZHANG, Zijin Gu, Zakaria Aldeneh, Navdeep Jaitly

Keywords: speech, tokenization, synthesis

TL;DR: We propose a simple and novel speech signal discretization method for speech model

Abstract:

Large language models have revolutionized natural language processing by leveraging self-supervised pretraining on vast textual data.
Inspired by this success, researchers have investigated complicated speech tokenization methods to discretize continuous speech signals so that language modeling techniques can be applied to speech data.
However, existing approaches either model semantic (content) tokens, potentially losing acoustic information, or model acoustic tokens, risking the loss of semantic (content) information.
Having multiple token types also complicates the architecture and requires additional pretraining.
Here we show that discretizing mel-filterbank channels into discrete intensity bins produces a simple representation (dMel), that performs better than other existing speech tokenization methods.
Using an LM-style transformer architecture for speech-text modeling, we comprehensively evaluate different speech tokenization methods on speech recognition (ASR) and speech synthesis (TTS).
Our results demonstrate the effectiveness of dMel in achieving high performance on both tasks within a unified framework, paving the way for efficient and effective joint modeling of speech and text. The code is available at anonymous_url, while generation samples are in the supplementary materials.

Primary Area: applications to computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5027