The Semantic Bottleneck: Leveraging Semantic Representations for Non-Invasive Speech Decoding

Gilad Landau, Dulhan Jayalath, Oiwi Parker Jones

Keywords: Brain Decoding, Semantics

TL;DR: A new method for decoding speech from brain based on semantic signal

Abstract:

Making non-invasive brain–computer interfaces (BCIs) for speech decoding a practical reality could substantially improve quality of life for many individuals. However, the low signal-to-noise ratio that characterizes non-invasive recording modalities remains a fundamental constraint and continues to hinder progress in this domain.

We introduce Brain2Semantics2Text, a new approach to non-invasive speech decoding. Rather than reconstructing speech directly at the word or phoneme level, the method targets high-level semantic representations. The approach is motivated by neuroscientific evidence that semantic information is represented across distributed cortical systems, and is enabled by recent progress in semantic embedding models that can be approximately inverted back into text.

The central principle of the method is a semantic bottleneck: neural activity is first mapped into a sentence-level semantic space, and only then reconstructed as text. We show that this approach improves BERTScore performance and compare our results against prior baselines.

Submission Number: 52