VoiceNoNG: High-Quality Speech Editing Model without Hallucinations

Sung-Feng Huang, Heng-Cheng Kuo, Zhehuai Chen, Xuesong Yang, Pin-Jui Ku, Ante Jukić, Chao-Han Huck Yang, Yu Tsao, Yu-Chiang Frank Wang, Hung-yi Lee, Szu-Wei Fu

Keywords: speech generative model, speech editing, neural codec, vector quantizer, deepfake detection

Abstract:

Currently, most advanced speech editing models are based on either neural codec
language models (NCLM) (e.g., VoiceCraft) or diffusion models (e.g., Voicebox).
Although NCLM can generate higher quality speech compared to diffusion models,
it suffers from a higher word error rate (WER) (Peng et al., 2024), calculated by
comparing the transcribed text to the input text. We identify that this higher WER
is due to attention errors (hallucinations), which make it difficult for NCLM to
accurately follow the target transcription. To maintain speech quality and address
the hallucination issue, we introduce VoiceNoNG, which combines the strengths of
both model frameworks. VoiceNoNG utilizes a latent flow-matching framework to
model the pre-quantization features of a neural codec. The vector quantizer in the
neural codec implicitly converts the regression problem into a token classification
task similar to NCLM. We empirically verified that this transformation is crucial
for enhancing the performance and robustness of the speech generative model. This
simple modification enables VoiceNoNG to achieve state-of-the-art performance
in both objective and subjective evaluations. Lastly, to mitigate the potential
risks posed by the speech editing model, we examine the performance of the
Deepfake detector in a new and challenging practical scenario. Audio examples
can be found on the demo page: https://anonymous.4open.science/w/NoNG-8004/

Primary Area: applications to computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 10751