DSA-Tokenizer: Disentangled Semantic-Acoustic Tokenization via Flow Matching-based Hierarchical Fusion

ACL ARR 2026 January Submission2314 Authors

Keywords: Speech Tokenizer, Speech LLM, Disentangle, Reconstrucion, Recombination

Abstract:

Speech tokenizers serve as the cornerstone of discrete Speech Large Language Models (Speech LLMs).
Existing tokenizers either prioritize semantic encoding, fuse semantic content with acoustic style inseparably, or achieve incomplete semantic-acoustic disentanglement. To achieve better disentanglement, we propose $\textbf{DSA-Tokenizer}$, which explicitly disentangles speech into discrete semantic and acoustic tokens via distinct optimization constraints. Specifically, semantic tokens are supervised by ASR to capture linguistic content, while acoustic tokens focus on mel-spectrograms restoration to encode style. To eliminate rigid length constraints between the two sequences, we introduce a hierarchical $\textbf{Flow-Matching}$ decoder that further improve speech generation quality. Furthermore, We employ a joint reconstruction-recombination training strategy to enforce this separation. DSA-Tokenizer enables high fidelity reconstruction and flexible recombination through robust disentanglement, facilitating controllable generation in speech LLMs. Our analysis highlights disentangled tokenization as a pivotal paradigm for future speech modeling. Audio samples are avaialble at $\url{https://anonymous.4open.science/w/DSA_Tokenizer_demo/}$. The code and model will be made publicly available after the paper has been accepted.

Paper Type: Long

Research Area: Speech Processing and Spoken Language Understanding

Research Area Keywords: Speech Recognition, Text-to-Speech and Spoken Language Understanding; Multimodality and Language Grounding to Vision, Robotics and Beyond

Contribution Types: Publicly available software and/or pre-trained models

Languages Studied: English

Submission Number: 2314