The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning
arxiv(2024)
Abstract
The past few years have produced a series of spectacular advances in the
decoding of speech from brain activity. The engine of these advances has been
the acquisition of labelled data, with increasingly large datasets acquired
from single subjects. However, participants exhibit anatomical and other
individual differences, and datasets use varied scanners and task designs. As a
result, prior work has struggled to leverage data from multiple subjects,
multiple datasets, multiple tasks, and unlabelled datasets. In turn, the field
has not benefited from the rapidly growing number of open neural data
repositories to exploit large-scale data and deep learning. To address this, we
develop an initial set of neuroscience-inspired self-supervised objectives,
together with a neural architecture, for representation learning from
heterogeneous and unlabelled neural recordings. Experimental results show that
representations learned with these objectives generalise across subjects,
datasets, and tasks, and are also learned faster than using only labelled data.
In addition, we set new benchmarks for two foundational speech decoding tasks.
Taken together, these methods now unlock the potential for training speech
decoding models with orders of magnitude more existing data.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined