Demonstration of Music Plus One - A Real-Time System for Automatic Orchestral Accompaniment

Christopher Raphael

Abstract: We demonstrate a system that creates a real-time accompaniment for a live musician performing a nonimprovisatory piece of music. The system listens to the live player by performing a hidden Markov model analysis of the player’s acoustic signal. A belief network uses this information, a musical score, and past rehearsals, to create a sequence of evolving predictions for future note-onsets in the soloist and accompaniment. These predictions are used to guide the timestretched resynthesis of prerecorded orchestral audio using a phase vocoder. Past work on musical accompaniment systems (Dannenberg 1984), (Dannenberg & Mukaino 1988), (Grubb & Dannenberg 1997), (Vercoe & Puckette 1985), (Baird, Blevins, & Zahler 1993), (Raphael 2001), (Raphael 2002), has focused mostly on generating MIDI or other sparselyparameterized accompaniments for live musicians. MIDI is is particularly well-suited to piano and percussion music since, in these cases, the MIDI representationcapturesmuch of the interpretative quality of the performance. Most other acoustic instruments are considerably less deterministic and their MIDI equivalents are currently much less convincing. We present here a system that generates a complete orchestral accompanimentthat follows a live player and learns and assimilates the player’s interpretation through a series of rehearsals. Unlike our previous efforts, this system creates the accompaniment by synthesizing, in real time, an orchestral accompaniment using an actual audio recording. Our system contains three separate modules called “Listen,” “Predict” and “Synthesize” which perform tasks analogous to the human’s hearing of the soloist, anticipating the future trajectory, and actual playing of the accompaniment. Listen is based on a hidden Markov model that tracks the soloist’s progress through the musical score. Each note in the scoreis modeledasa series ofstates, whilethe notemodels are chained together in left-to-right fashion to form the hiddenprocess. Listen can be used “off-line,”to estimate the onset timesof each note in the score giventhe entireacoustic data file, or “on-line,” by delivering real-time onset time estimates. In the on-line version these estimates are delivered