Acoustic inspired brain-to-sentence decoder for logosyllabic language
image: (A) After the electrodes were implanted, participants took part in the study in the wheelchair, reading sentences given visual cues. Audio data were collected using a directional microphone, and neural signals were simultaneously recorded using a multi-channel electrophysiological recording system. (B) Two tasks designed for collecting training used data (upper panel, 407 syllables cover almost all Mandarin Chinese characters) and evaluating used data (lower panel, 100 sentences ranging from 2 to over 10 characters in length). (C) The distribution of selected decoding channels across participants is illustrated on the standard Montreal Neurological Institute (MNI) template brain. Each participant’s channels are denoted by distinct colors, with directional indicators provided for clarity. (D) A 3-phase decoder, including channel selection, syllable element prediction, and language model correction, is designed to output whole Mandarin Chinese sentences by decoding speech-related sEEG signals. CNN, convolutional neural networks; MLP, multilayer perceptron.
Credit: Chen Feng, Westlake University.
Recent advances in brain–computer interfaces (BCIs) have demonstrated the potential to decode language from brain activity into sound or text, which has predominantly focused on alphabetic languages, such as English. However, logosyllabic languages, such as Mandarin Chinese, present marked challenges for establishing decoders that cover all characters, due to its unique syllable structures, extended character sets (e.g., over 50,000 characters for Mandarin Chinese), and complex mappings between characters and syllables, thus hindering practical applications. “Considering the fact that the pronunciation of Mandarin character is mostly detached from its glyphs, we seek a viable solution to bypass the curse of logographic systems via decoding the pronunciation of characters.” said the author Chen Feng, a researcher at Westlake University, “We leverage the acoustic features of Mandarin Chinese syllables, constructing prediction models for syllable components (initials, tones, and finals), and decode speech-related stereoelectroencephalography (sEEG) signals into coherent Chinese sentences.”
The model follows a three-stage pipeline: (1) speech-related cortical and thalamic sEEG channels are selected while signals from visual cortex and white matter are discarded; (2) each syllable segment is fed into three parallel convolutional branches that respectively predict the initial consonant, tone and rhyme cluster. The consonant branch first maps neural features to articulatory attributes—place and manner of articulation, aspiration and voicing—and refines them with a graph neural network, whereas the tone and rhyme branches use neural-acoustic regularization to align neural activity with acoustic cues such as fundamental frequency and formant contours for sharper discrimination. (3) The concatenated consonant-tone-rhyme probability sequence for the whole sentence is passed to a language model: a 5-gram model performs coarse pruning, after which Chinese-LLaMA resolves homophones, corrects errors and maps the sequence to the most probable Chinese characters, enabling a system trained on only 407 base syllables to transcribe arbitrary Mandarin sentences from brain signals.
Overall, this study demonstrates a novel approach to offline decoding Mandarin Chinese, a complex logosyllabic language, from brain activity into coherent sentences. By leveraging unique acoustic features of Mandarin syllables and integrating advanced neural network models, authors achieved a median character accuracy of 71.00%, with 30.00% of sentences decoded completely accurately. This method highlights the critical role of both cortical and subcortical brain signals and the importance of acoustic-related features in enhancing prediction accuracy. “In future, our research will focus on improving cross-day model transfer, online decoding performance, and optimizing data utilization from multiple subjects and sessions.” said Chen Feng.
Authors of the paper include Chen Feng, Lu Cao, Di Wu, En Zhang, Ting Wang, Xiaowei Jiang, Jinbo Chen, Hui Wu, Siyu Lin, Qiming Hou, Junming Zhu, Jie Yang, Mohamad Sawan, and Yue Zhang.
This project was funded by STI2030-Major Project (2022ZD0208805), National Natural Science Foundation of China (grant no. 623B2085), and “Pioneer” and “Leading Goose” R&D Program of Zhejiang (2024C03002).
The paper, “Acoustic Inspired Brain-to-Sentence Decoder for Logosyllabic Language” was published in the journal Cyborg and Bionic Systems on Apr. 29, 2025, at DOI: 10.34133/cbsystems.0257.