Improving Speech Enhancement via Event-Based Query

Yifei Xin, Xiulian Peng, Yan Lu

Abstract: Existing deep learning based speech enhancement (SE) methods either use blind end-to-end training or explicitly incorporate speaker embedding or phonetic information into the SE network to enhance speech quality. In this paper, we perceive speech and noises as different types of sound events and propose an event-based query method for SE. Specifically, speech embeddings that can discriminate speech from noises are first pre-trained with the sound event detection (SED) task. The embeddings are then clustered into fixed golden speech queries, i.e., general but representative speech embeddings, on a diverse clean speech dataset to assist the SE network. The golden speech queries can be obtained offline and generalizable to different SE datasets and networks. Therefore, little extra complexity is introduced and no enrollment is needed for each speaker. Experimental results show that the proposed method yields significant gains compared with baselines and the golden queries are well generalized to different datasets.