Single-Stage, Low-Latency, and Intelligible Brain-to-Speech Synthesis

Abstract

The loss of speech limits communication for individuals with paralysis. Direct neural-to-speech synthesis is challenging due to the limited availability of neural data for training speech brain–computer interfaces. Most existing systems rely on cascaded neural-to-text-to-speech pipelines, which increase inference latency and propagate errors across stages. We present Brain2Speech-Net, a single-stage neural-to-speech generation framework without intermediate text decoding. We use a differentiable phoneme bottleneck and a deep-HMM alignment mechanism to map long neural recordings into the latent space of a pre-trained text-to-speech (TTS) model, enabling high-quality speech synthesis. On an intracortical dataset, Brain2Speech-Net achieves strong intelligibility with low latency in objective and listening tests. Unlike cascaded systems that have high latency and direct speech unit-based models that lack intelligibility, it delivers intelligible speech and real-time performance.

Model Overview

Contact & Acknowledgements