Towards Linguistic Neural Representation Learning and Sentence Retrieval from Electroencephalogram Recordings
arxiv(2024)
摘要
Decoding linguistic information from non-invasive brain signals using EEG has
gained increasing research attention due to its vast applicational potential.
Recently, a number of works have adopted a generative-based framework to decode
electroencephalogram (EEG) signals into sentences by utilizing the power
generative capacity of pretrained large language models (LLMs). However, this
approach has several drawbacks that hinder the further development of
linguistic applications for brain-computer interfaces (BCIs). Specifically, the
ability of the EEG encoder to learn semantic information from EEG data remains
questionable, and the LLM decoder's tendency to generate sentences based on its
training memory can be hard to avoid. These issues necessitate a novel approach
for converting EEG signals into sentences. In this paper, we propose a novel
two-step pipeline that addresses these limitations and enhances the validity of
linguistic EEG decoding research. We first confirm that word-level semantic
information can be learned from EEG data recorded during natural reading by
training a Conformer encoder via a masked contrastive objective for word-level
classification. To achieve sentence decoding results, we employ a training-free
retrieval method to retrieve sentences based on the predictions from the EEG
encoder. Extensive experiments and ablation studies were conducted in this
paper for a comprehensive evaluation of the proposed approach. Visualization of
the top prediction candidates reveals that our model effectively groups EEG
segments into semantic categories with similar meanings, thereby validating its
ability to learn patterns from unspoken EEG recordings. Despite the exploratory
nature of this work, these results suggest that our method holds promise for
providing more reliable solutions for converting EEG signals into text.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要