Karaoke of Dreams: A multi-modal neural-network generated music experience

Sophia Huiwen Sun

Abstract: Karaoke of Dreams (KoD) is a deep learning karaoke environment that generates songs and video based on user input of song titles. KoD is a multi-dimensional machine learning matrix operating in the aural dimension on harmony, melody, lyrics and style. In the musical dimension it plays 5-track pop songs generated by a Generative Adversarial Network (GAN) rendered through a Pure Data patch. In the visual dimension it operates on images tied to words using attention GANs. In the linguistic dimension it is generated by GPT-2 fine-tuned on pop song lyrics. In the spatial dimension it is a 4-fold rotational tesseract, outlining 3D space. The human element is the participant’s voice and performance of the generated concert experience. KoD envelops the singer like an AI womb, a safe, warm, pulsating center allowing the singer to pour their heart out.