Translating neural signals to text using a Brain-Computer Interface

Janaki Sheth, Ariel Tankus, Michelle Tran, Nader Pouratian, Itzhak Fried, William Speier

TL;DR: We present an open-loop brain-machine interface whose performance is unconstrained to the traditionally used bag-of-words approach.

Keywords: Brain-Computer interface, Speech decoding, Neural signal processing

Abstract: Brain-Computer Interfaces (BCI) may help patients with faltering communication abilities due to neurodegenerative diseases produce text or speech by direct neural processing. However, their practical realization has proven difficult due to limitations in speed, accuracy, and generalizability of existing interfaces. To this end, we aim to create a BCI that decodes text directly from neural signals. We implement a framework that initially isolates frequency bands in the input signal encapsulating differential information regarding production of various phonemic classes. These bands form a feature set that feeds into an LSTM which discerns at each time point probability distributions across all phonemes uttered by a subject. Finally, a particle filtering algorithm temporally smooths these probabilities incorporating prior knowledge of the English language to output text corresponding to the decoded word. Further, in producing an output, we abstain from constraining the reconstructed word to be from a given bag-of-words, unlike previous studies. The empirical success of our proposed approach, offers promise for the employment of such an interface by patients in unfettered, naturalistic environments.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/translating-neural-signals-to-text-using-a/code)