DualCodec: A Low-Frame-Rate, Semantically-Enhanced Neural Audio Codec for Speech Generation

OpenReview Anonymous Preprint Submission653 Authors

Keywords: neural audio codec, speech generation, low-frame-rate

Abstract: Neural audio codecs form the foundational building blocks for language model (LM)-based speech generation. Typically, there is a trade-off between frame rate and audio quality. This study introduces a low-frame-rate, semantically enhanced codec model. Existing approaches distill semantically rich self-supervised (SSL) representations into the first-layer codec tokens. This work proposes DualCodec, a dual-stream encoding approach that integrates SSL and waveform representations within an end-to-end codec framework. In this setting, DualCodec enhances the semantic information in the first-layer codec and enables the codec system to maintain high audio quality while operating at a low frame rate. Note that a low-frame-rate codec improves the efficiency of speech generation. Experimental results on audio codec and speech generation confirm the effectiveness of the proposed DualCodec compared to state-of-the-art codec systems, such as Mimi Codec, DAC, Encodec, and SpeechTokenizer. Demos are available at: https://dualcodec.github.io

Submission Number: 653