Improving Target Sound Extraction via Disentangled Codec Representations with Privileged Knowledge Distillation

Dail Kim, Joon-Hyuk Chang

Keywords: Target Sound Extraction, Privileged Knowledge distillation, Disentangled Representation Learning, Neural Audio Codec, Feature-level Knowledge Distillation

TL;DR: This paper proposes DCKD, a privileged knowledge distillation framework for target sound extraction that regulates the amount and flow of target information via neural codec and disentangled representation learning.

Abstract: Target sound extraction aims to isolate target sound sources from an input mixture using a target clue to identify the sounds of interest. To address the challenge posed by the wide variety of sounds, recent work has introduced privileged knowledge distillation (PKD), which utilizes privileged information (PI) about the target sound, available only during training. While PKD has shown promise, existing approaches often suffer from overfitting of the teacher model for the overly rich PI and ineffective knowledge transfer to the student model. In this paper, we propose Disentangled Codec Knowledge Distillation (DCKD) to mitigate these issues by regulating the amount and the flow of target sound information within the teacher model. We begin by extracting a compressed representation of the target sound using a neural audio codec to regulate the amount of PI. Disentangled representation learning is then applied to remove class information and extract fine-grained temporal information as PI. Subsequently, an n-hot vector as the class information and the class-independent PI are used to condition the early and later layers of the teacher model, respectively, forming a regulated coarse-to-fine target information flow. The resulting representation is transferred to the student model through feature-level knowledge distillation. Experimental results show that DCKD consistently improves existing methods across model architectures under the multi-target selection condition.

Primary Area: Applications (e.g., vision, language, speech and audio, Creative AI)

Submission Number: 28819