VChangeCodec: A High-efficiency Neural Speech Codec with Built-in Voice Changer for Real-time Communication
Keywords: Real-time communication, Neural Speech codec, Voice conversion
Abstract: Neural speech codecs (NSCs) enable high-quality real-time communication (RTC)
at low bit rates, making them efficient for bandwidth-constrained environments.
However, customizing or modifying the timbre of transmitted voices still relies on
separate voice conversion (VC) systems, creating a gap in fully integrated systems
that can simultaneously optimize efficient transmission and streaming VC with no
additional latency. In this paper, we propose a high-efficiency VChangeCodec,
which integrates the Voice Changer model directly into the speech Codec. This
design seamlessly switches between the original voice mode and customized voice
change mode in real-time. Specifically, leveraging the target speaker’s embedding,
we incorporate a lightweight causal projection network within the encoding module
of VChangeCodec to adapt timbre at the token level. These adapted tokens are
quantized and transmitted to the decoding module, to generate the converted speech
of the target speaker. The integrated framework achieves an ultra-low latency of
just 40 ms and requires fewer than 1 million parameters, making it ideal for RTC
scenarios such as online conferencing. Our comprehensive evaluations, including
subjective listening tests and objective performance assessments, demonstrate that
VChangeCodec excels in timbre adaptation capabilities compared to state-of-the-art (SOTA) VC models. We are confident that VChangeCodec provides an efficient
and flexible framework for RTC systems, tailored to specific operator requirements.
Supplementary Material: zip
Primary Area: applications to computer vision, audio, language, and other modalities
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 10137
Loading