CSLNSpeech: Solving the extended speech separation problem with the help of Chinese sign language

Jiasong Wu, Xuan Li, Taotao Li, Fanman Meng, Youyong Kong, Guanyu Yang, Lotfi Senhadji, Huazhong Shu

Abstract: Highlights•A new multi-modalities dataset, consisting of audio, face, and sign language, named Chinese Sign Language News Speech (CSLNSpeech) is proposed.•A common benchmark on CSLNSpeech for solving multi-modalities speech separation is proposed.•Exploring the influence of sign language modality in speech separation for the first time.•Performing very favorably against other traditional training strategies.