Decoding spoken English from intracortical electrode arrays in dorsal precentral gyrus

Guy H Wilson, Sergey Stavisky, Francis R Willett, Donald Avansino, jessica kelemen, Leigh Hochberg, Jaimie M. Henderson, shaul druckmann, Krishna V. Shenoy

Abstract:

Objective. To evaluate the potential of intracortical electrode array signals for brain-computer
interfaces (BCIs) to restore lost speech, we measured the performance of decoders trained to
discriminate a comprehensive basis set of 39 English phonemes and to synthesize speech sounds
via a neural pattern matching method. We decoded neural correlates of spoken-out-loud words in
the ‘hand knob’ area of precentral gyrus, a step toward the eventual goal of decoding attempted
speech from ventral speech areas in patients who are unable to speak. Approach. Neural and audio
data were recorded while two BrainGate2 pilot clinical trial participants, each with two
chronically-implanted 96-electrode arrays, spoke 420 different words that broadly sampled English
phonemes. Phoneme onsets were identified from audio recordings, and their identities were then
classified from neural features consisting of each electrode’s binned action potential counts or
high-frequency local field potential power. Speech synthesis was performed using the
‘Brain-to-Speech’ pattern matching method. We also examined two potential confounds specific to
decoding overt speech: acoustic contamination of neural signals and systematic differences in
labeling different phonemes’ onset times. Main results. A linear decoder achieved up to 29.3%
classification accuracy (chance = 6%) across 39 phonemes, while an RNN classifier achieved
33.9% accuracy. Parameter sweeps indicated that performance did not saturate when adding more
electrodes or more training data, and that accuracy improved when utilizing time-varying
structure in the data. Microphonic contamination and phoneme onset differences modestly
increased decoding accuracy, but could be mitigated by acoustic artifact subtraction and using a
neural speech onset marker, respectively. Speech synthesis achieved r = 0.523 correlation between
true and reconstructed audio. Significance. The ability to decode speech using intracortical
electrode array signals from a nontraditional speech area suggests that placing electrode arrays in
ventral speech areas is a promising direction for speech BCIs.