The pipeline of the brain-to-sentence decoder designed for logosyllabic language decoding. (IMAGE)
Caption
(A) After the electrodes were implanted, participants took part in the study in the wheelchair, reading sentences given visual cues. Audio data were collected using a directional microphone, and neural signals were simultaneously recorded using a multi-channel electrophysiological recording system. (B) Two tasks designed for collecting training used data (upper panel, 407 syllables cover almost all Mandarin Chinese characters) and evaluating used data (lower panel, 100 sentences ranging from 2 to over 10 characters in length). (C) The distribution of selected decoding channels across participants is illustrated on the standard Montreal Neurological Institute (MNI) template brain. Each participant’s channels are denoted by distinct colors, with directional indicators provided for clarity. (D) A 3-phase decoder, including channel selection, syllable element prediction, and language model correction, is designed to output whole Mandarin Chinese sentences by decoding speech-related sEEG signals. CNN, convolutional neural networks; MLP, multilayer perceptron.
Credit
Chen Feng, Westlake University.
Usage Restrictions
News organizations may use or redistribute this image, with proper attribution, as part of news coverage of this paper only.
License
Licensed content