Perceiving and Learning Color as Sound in Virtual Reality

Ethan Seefried, Jessica Bahny, Changsoo Jung, Videep Venkatesha, Mariah Bradford, Nathaniel Blanchard, Mohammed Safayet Arefin

Abstract: Virtual reality (VR) is an emerging technology with the capacity to enhance users’ perceptual skills. People with visual impairments, such as color blindness and low vision, often struggle to perceive colors accurately in VR settings. This paper introduces a novel technique that translates sound into color perception within a VR environment. The approach involves mapping seven primary colors from the visible spectrum onto specific piano notes in the major scale. Three experimental tasks—ordering, selection, and the Stroop test—were conducted. Results from the user study revealed promising kappa values of 0.73 and 0.52 for the selection task and Stroop test, respectively, indicating successful learning and understanding of the color-sound associations. This research opens new possibilities for sound-based color perception to enhance accessibility and therapeutic use in VR.